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ABSTRACT 


The  Army  is  currently  developing  the  Performance  Risk  Assessment  Group 
(PRAG)  program  as  a  measure  to  maximize  the  use  of  past  performance  in  die 
source  selection  process.  The  use  of  past  performance  is  becoming  increamn^y 
important  as  the  defense  budget  gets  smaller  and  the  use  of  commercially  accqited 
business  practices  becomes  more  emphasized.  A  program  to  evaluate  past 
performance  can  better  ensure  the  Army  gets  a  quality  product  at  a  tidr  price.  The 
Army  Material  Command  developed  tiie  "Past  Performance  in  Source  Selection; 
An  Evaluation  Guide"  to  assist  buying  commands  in  implementing  the  PRAG 
effort.  This  thesis  analyses  that  guide,  identifies  its  strengths  and  weaknesses,  and 
makes  recommendations  to  maximize  its  use. 
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Z.  ZMTRODDCTZOV 


A.  BACKGROOm) 

As  the  situation  in  Eastern  Bloc  Nations  continues  to 
chcuige  and  the  cold  war  draws  to  an  end,  we  are  e3q>erienclng 
extraordinary  changes  in  the  Department  of  Defense.  Not  only 
are  the  Services  dommsizing,  but  their  procurement  budgets  are 
getting  smaller  as  well.  This  maJces  the  challenge  of 
equipping  our  forces  with  the  best  possible  hardware,  tough 
euid  demanding.  The  only  way  we  will  meet  this  requirement,  in 
the  environment  we  face,  is  to  inqplement  measures  that  ensure 
every  procurement  dollar  spent  maximizes  our  buying  effort. 

One  way  of  ensuring  procurement  dollars  are  well  spent  is 
to  prevent  award  of  contracts  to  contractors  whose  perfornance 
is  risky.  This  is  the  responsibility  of  the  contracting 
officer  who  now  has  the  Performance  Risk  Assessment  Group 
(PRAG)  to  assist  him/her  with  the  task.  The  PRAG's  job  is  to 
assess  the  quality  of  a  contractor's  past  performance  and  how 
that  may  contribute  to  performance  on  a  future  contract  for 
which  they  have  submitted  a  proposal. 

The  PRAG  is  a  requirement  developed  by  the  Army  Material 
Comnand  (AMC)  for  eULl  subordinate  buying  activities. 
Recently,  AMC  Pan^hlet  715-3,  Volume  4;  "Past  Perfornance  in 
Source  Selection;  An  Evaluation  Guide",  was  published  to 
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assist  buying  consnands  In  lnplementlng  this  effort.  Ilils 
document  Is  the  most  comprehensive  tool  available  for 
maximizing  the  use  of  past  performance  In  the  source  selection 
process.  Performance  measures  to  be  evaluated,  procurement 
procedures  to  be  used  euid  questions  regarding  recency  and 
relevancy  are  all  In^orteuit  In  In^lementlng  a  successful  PRAG 
program. 

B.  OBJECTIVES 

As  a  result  of  Increased  en^hasls  on  best  value 
procurements,  awards  based  on  criteria  other  than  price. 
Increased  congressional  oversight  and  the  push  for  using 
commercially  accepted  business  practices,  the  Army  has 
eirphaslzed  the  Importance  of  past  performance.  In  order  to 
establish  guidelines  for  the  use  of  past  performance,  the  Am^ 
Material  Command  published  "Past  Performance  In  Source 
Selection;  an  Evaluation  Guide”,  hereafter  referred  to  as  the 
PPG. 

This  thesis  Is  an  Independent  analysis  of  the  Amy's  PPG. 
To  m^lmlze  efforts  In  using  past  performance  In  the  source 
selection  process,  a  viable  guide  must  exist  to  ensure  a 
degree  of  consistency  among  the  practices  used  by  all 
sxibordlnate  buying  activities  In  AMC.  The  guide  must 
represent  alternatives  available  for  maximizing  the  use  of 
past  performance.  Additionally,  the  guide  must  discuss  the 
appropriate  criteria  that  measure  the  quality  of  contractor's 
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past  perf oirmance .  It  must  also  provide  guidamce  on  the  use  of 
past  performance  data  and  whether  they  are  relevant  to  a 
current  contract  or  recent  enough  to  provide  a  fair  assessment 
of  a  contractor's  expected  perf ormeuice .  After  a  thorough 
analysis  of  the  PPG,  appropriate  recommendations  are  made  to 
mcucimize  its  effectiveness  without  being  so  specific  that  it 
does  not  allow  flexibility  within  the  buying  commands  to 
accommodate  their  unique  requirements. 

C.  THE  RESEARCH  QUESTIONS 

The  primary  research  question  is:  How  Ccui  the  Army 

maximize  its  use  of  past  performance  in  the  source  selection 
process?  The  subsidiary  research  questions  are: 

1.  Nhat  performance  measures  should  be  evaluated  for  source 
selection? 

2.  What  problems  have  been  encountered  with  the  previous 
use  of  past  performance  for  source  selection? 

3 .  What  information  is  available  for  use  by  the  Performance 
Risk  Assessment  Group? 

4.  What  special  conditions  exist  in  the  source  selection 
process  when  past  performance  is  used  as  an  evaluation 
criterion? 

5 .  How  can  past  performance  be  used  under  sealed  bidding 
procedures? 

6.  What  modifications  are  necessary  in  order  to  maximize 
the  use  of  the  "Army's  Past  Performance  in  Source  Selection 
Evaluation  Guide"? 
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D.  SCOPE,  LIMITATIONS  AND  ASSUMPTIONS 

The  scope  of  this  thesis  is  to  evaluate  performance 
measures  and  procedures  that  are  In^ortant  to  the  source 
selection  process  in  sealed  bidding  and  as  an  evaluation 
criterion  in  con^etitive  proposals.  This  thesis  includes  both 
Government  and  industry  procedures.  Emphasis  is  placed  on  the 
issues  associated  with  the  procurement  of  aviation  conponents 
in  the  Army,  Navy,  and  Air  Force;  contracting  for  services  is 
not  addressed.  Findings  were  compared  to  the  Army's  PPG  to 
determine  the  strengths  and  weaknesses  of  that  dociunent. 
Recommendations  have  been  developed  to  modify  the  PPG,  meiking 
it  a  more  useful  tool  for  meocimizing  the  use  of  past 
performance . 

E.  METHODOLOGY 

Two  methods  were  used  to  collect  data  and  answer  the 
research  questions.  Data  were  collected  from  secondary 
sources  to  gain  an  understanding  of  PRAG  and  the  initiatives 
used  in  the  other  Services.  Telephone  interviews  were  used  to 
gain  an  understanding  of  the  initiatives  used  in  the  Services; 
including  their  strengths  and  weaknesses .  Interviews  involved 
personnel  from  the  Army's  Aviation  and  Troop  Command  (ATCOM) , 
the  Air  Force's  Aeronautical  Systems  Division  (ASD) ,  and  the 
Navy's  Aviation  Supply  Office  (ASO) .  Additionally,  telephone 
interviews  were  conducted  with  several  companies  to  get  eui 
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idea  of  the  initiatives  currently  being  used  by  industry  in 
using  past  performance  in  the  selection  of  suppliers. 

P.  ORGANIZATION  OF  TEN  STUDY 

The  remainder  of  the  thesis  is  organized  into  four 
additional  chapters.  Chapter  II  describes  why  past 
performance  has  become  a  procurement  issue.  A  summary  of  the 
Army's  PRA6  concept  ouid  initiatives  developed  by  the  other 
Services  are  also  presented.  Additional  information  includes: 
a  discussion  of  the  source  selection  process  in  sealed  bidding 
and  competitive  negotiations  and  the  determination  of 
contractor  responsibility.  Chapter  III  focuses  on  the 
presentation  and  analysis  of  data  received  during  telephone 
interviews  with  industry.  The  information  from  Chapters  II 
and  III,  feedback  from  interviews  with  military  organizations, 
and  additional  second  source  data  were  used  to  evaluate  the 
effectiveness  of  the  Army's  PPG.  This  analysis  is  presented 
in  Chapter  IV.  Chapter  V,  conclusions  and  recommendations, 
develops  findings  concerning  the  study  and  makes 
recommendations  for  improving  the  use  of  PRAG  and  the  PPG. 
Additionally,  answers  to  the  research  (questions  cuid 
suggestions  for  future  research  are  presented. 
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ZZ.  BAOCOSOmiD  LZTISATDSX 


A.  ZMPORTANCB  OF  PAST  PIRFOSICAHCB 

The  President's  Blue  Ribbon  Comnlsslon  on  Defense 

Management,  June  1986,  better  known  as  the  Packard  Commission, 

provided  recommendations  to  Improve  the  acquisition 

organization  and  procedures  used  by  the  Department  of  Defense 

(DoD)  .  The  commission  recognized  the  lnportcuice  of  using 

commercially  accepted  business  practices  as  a  method  of 

streamlining  DoD  procurement  [Ref.  18 :p.  62]: 

Federal  law  and  DoD  regulations  should  provide  for 
substantially  Increased  use  of  commercial  style 
conpetltlon,  emphasizing  quality  and  established 
perfozmance  as  well  as  price. 

Defense  procurements  have  typically  concentrated  on  price 
as  a  primary  determinate  for  source  selection;  generally 
Ignoring  other  attributes  that  may  prove  Inportcmt.  However, 
price  should  not  be  the  only  determinate  for  source  selection, 
especially  for  the  procurement  of  complex  systems.  [Ref.  18 :p. 
62] 

The  consideration  of  past  performance  as  an  evaluation 
criterion  Is  a  subset  of  a  "best  value”  procurement.  An 
underlying  principle  of  best  value  Is  that  the  Government 
should  use  sound  business  practices  when  procuring  goods  euid 
services;  allowing  contract  award  based  on  criteria  other  than 
price  [Ref.  14 :p.  3] . 
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The  Federal  Acquisition  Regulation  (FAR)  states  [Ref. 
lOrSubpart  15.605]: 

. .  .while  the  lowest  total  cost  to  the  Government  Is 
properly  the  deciding  factor  In  many  source  selections.  In 
certain  acquisitions  the  Government  may  select  the  source 
whose  proposal  offers  the  greatest  value  to  the  Government 
In  terms  of  performcuice  and  other  factors. 


The  following  position  by  an  expert  on  DoD  procurement  policy 
Is  representative  of  those  advocating  the  use  of  commercial 
practices  to  achieve  the  best  overall  value  In  source 
selection  [Ref.  14 :p.  3] : 

The  overwhelming  majority  of  items  could  be  purchased  in 
a  commercial  fashion. . . .This  would  entail  allowing  "best 
value"  judgment  by  the  government  buyer  and  no  protest  by 
the  losers  (except  In  cases  of  Illegality)  .  It  also  means 
that  as  long  the  prices  being  bid  were  established  by  the 
commercial  marlcet,  the  supplier  would  not  be  required  to 
provide  a  detailed  price  breakdown. . . (again  as  In  the 
commercial  world) .  Finally,  and  perhaps  the  most 
In^ortant,  the  prior  performance  of  a  supplier  should  play 
a  major  role  In  the  evaluation.  A  supplier  that  has  done 
a  good  job  should  be  given  favoreUsle  treatment,  and  one 
that  has  done  a  poor  job  should  not  be  considered  for  the 
next  procurement. 


Current  regulations  contain  little  guidance  on  best  value 
procurements  and  the  use  of  past  performance  as  a  criterion 
for  source  selection.  Although  recent  litigation  and  agency 
documents  have  resolved  some  past  performance  Issues, 
Information  regarding  recent  developments  Is  not  readily 
available  in  a  single  source.  [Ref.  14 :p.  4]  Additionally, 
legislation  has  led  to  some  confusion  as  to  the  Intent  of  such 
statutes.  The  Con^etition  in  Contracting  Act  (CICA)  recjuires 
full  and  open  competition,  but  many  procurement  officials  have 
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Interpreted  this  to  meeui  that  the  Government  is  required  to 
buy  from  the  lowest  offeror.  However,  CICA  does  recognize  the 
Ixiportance  of  con^etltlon  and  award  based  on  am  overall 
assessment  of  a  contractor's  proposal,  not  just  the  price. 
[Ref.  18 :p.  63] 

EstcUallshed  performeuice  Is  am  Importauit  attribute  for  the 
selection  of  suppliers  In  commercial  Industries.  Typically,  r 
list  of  qualified  suppliers,  those  with  a  record  of  delivering 
quality  products  auid  maintaining  high  steuidards  of 
performauice.  Is  maintained.  [Ref.  18 :p.  62]  The  same  practice 
could  benefit  Government  as  well.  The  Department  of  Defense 
directed  similar  action  through  DoD  Directive  5000.2  which 
states  [Ref.  24:p.  6-P-2] : 

Contractor  past  history  of  providing  quality  products  amd 
services  shall  be  considered  during  the  evaluation  of 
proposals  from  potential  contractual  sources.  Objective 
contractor  quality  data  shall  be  collected  amd  maintained 
for  this  purpose. 

B.  ARMT'S  PERFORMANCE  RISK  ASSESSMENT  GROUP 
1.  Background 

The  Army  Is  currently  In^lementlng  efforts  to  evaluate 
past  performance  through  the  Performamce  Risk  Assessment  Group 
(PRAG)  program.  The  acquisition  community  has  always 
recognized  the  Importamce  of  past  performance  In  the  selection 
of  quality  contractors.  [Ref.  2:p.  1] 

The  PRAG  program  has  two  unique  characteristics. 
First,  It  relies  on  Information  outside  the  offeror's  proposal 
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to  evaluate  past  performamce.  Second,  It  relies  on  the 
Independent  evaluation  of  past  performance  by  the  PRA6  which 
Is  separate  from  the  proposal  evaluation  team.  [Ref.  2:p.  2] 

The  procedures  outlined  In  the  PPG  apply  to 
conpetltlvely  negotiated  contracts  where  aiiard  Is  based  on 
criteria  other  than  price;  known  as  "best  value"  procurements. 
The  Office  of  Federal  Procurement  Policy  (OPPP) ,  Policy  Letter 
92-5,  requires  the  evaluation  of  past  performcuice  on  all 
con^etltlvely  negotiated  contracts  that  exceed  $100,000.  [Ref. 
26  :p.  2]  However,  the  procedures  outlined  In  the  PPG  are  only 
required  for  procurements  greater  than  $10,000,000  [Ref.  2:p. 
2]  . 

The  PRAG  approach  differs  somewhat  from  pre-a«rard 
surveys.  While  a  pre-award  survey  provides  a  "yes/no", 
"pass/fall"  Input  to  the  responsibility  determination,  the 
PRAG  offers  a  con^aratlve  approach  Identifying  the  degree  of 
risk  associated  with  an  offeror's  proposal.  It  provides  the 
likelihood  of  success  of  a  contractor.  The  pre- award  survey 
euid  past  performauice  evaluation  are  designed  to  con^lement 
each  other.  [Ref.  2;p.  3] 

2.  PRAG  Structure,  Coiqposltlon  and  Evaluation 

The  PRAG  is  responsible  for  determining  the 
performance  risk  of  a  contractor.  The  PRAG  prepares  a  report 
describing  the  strengths  and  weaknesses  of  an  offeror's  past 
perfomieuice  [Ref.  2:p.  3]. 
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Bach  subordloate  buying  activity  determines  the 
composition  and  size  of  the  PRAG  based  on  the  size,  nature  and 
complexity  of  a  procurement.  Memibershlp  should  include 
Individuals  with  technical,  cost  auid  procurement  esqpertlse. 
The  primary  information  used  by  the  PRAG  Is  gathered  through 
questionnaires  and  telephone  Interviews.  The  Importance  of 
this  Information  Is  critical;  It  Is  the  basis  upon  which  em 
overall  assessment  of  past  performance  Is  made.  [Ref.  2:p.  4] 

The  amount  of  weight  given  to  past  performance  should 
be  substantial  enough  that  It  acts  as  a  valid  discriminator. 
The  Government  can  evaluate  both  experience  euid  past 
performance.  It  Is  Important,  though,  not  to  evaluate  this 
factor  twice.  If  past  perfoimiance  Is  evaluated  separately,  it 
should  not  be  evaluated  again  as  an  aspect  of  experience.  It 
would,  however,  be  proper  to  evaluate  personnel  esqperlence  and 
past  performance  or  personnel  experience  euid  company 
experience.  [Ref.  2:p.  5] 

The  PRAG  Is  designed  to  operate  independently.  It  may 
operate  separately  from  the  Source  Selection  Evaluation  Board 
(SSEB) ,  reporting  directly  to  the  Source  Selection  Advisory 
Council  (SSAC)  ,  or  Ceui  operate  as  a  subgroup  of  the  SSEB  eUid 
report  to  the  SSEB  chairperson.  A  PRAG  assessment  plan  should 
be  Included  as  part  of  the  source  selection  plcUi.  [Ref.  2:p. 
4] 
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3.  Sub-fmotors 

The  sub- factors  to  be  used  by  the  PRA6  do  not  have  to 
mirror  those  used  by  the  proposal  evaluation  team.  Ideally, 
the  PRAG  will  evaluate  am  offeror's  record  for  on  time 
delivery,  technical  quality  amd  cost  control.  OFPP  Policy 
Letter  92-5  suggests  several  other  areas  to  evaluate:  record 
of  conforming  to  specifications,  stamdards  of  good 
workmamship,  record  of  containing  forecasted  costs  on  other 
cost -reimbursement  type  contracts,  adherence  to  schedules, 
administrative  aspects  of  performance,  cooperative  behavior 
history,  commitment  to  customer  satisfaction  and  a  concern  for 
the  interest  of  the  customer.  Only  those  contracts  relevant 
to  the  contract  for  which  the  offeror  has  submitted  a  proposal 
should  be  addressed.  [Ref.  26 :p.  1] 

4 .  Rating  Categories 

The  following  definitions  of  performance  risks  are 
used  to  describe  the  results  of  the  PRAG  assessment  [Ref.  2:p. 
6]  ; 


1.  High  Perfoxmamce  Risk  -  Based  on  the  offeror's 
perfomance  record,  significant  doubt  exists  that  the 
offeror  will  successfully  perform  the  required  effort. 

2.  Moderate  Performance  Risk  -  Based  on  the  offeror's 
performance  record,  some  doubt  exists  that  the  offeror  will 
successfully  perform  the  required  effort. 

3.  Low  Performance  Risk  -  Based  on  the  offeror's 
perfomance  record,  little  doubt  exists  that  the  offeror 
will  successfully  perform  the  required  effort. 
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4 .  Unknown  Performance  Risk 
IdentlfloUsle. 


No  performance  record 


Contractors  can  be  remk  ordered  in  each  category  to  give  a 
better  con^rative  amalysls.  Contractors  with  am  \mknown 
performamce  risk  are  treated  as  neutral  amd  no  weight  is  given 
to  past  performance.  However,  in  circiunstances  where  a  proven 
perf oinnamce  record  is  required,  a  high  or  moderate  performamce 
risk  may  be  assigned.  The  solicitation  must  clearly  state 
that  the  PRAG  will  evaluate  experience  as  well  as  past 
performamce  amd  that  the  lack  of  experience  could  result  in  a 
high  or  moderate  risk  rating.  [Ref.  2:p.  6] 

5 .  Solicitation 

When  am  assessment  of  past  performance  will  be  used  as 
an  evaluation  criterion  ,  the  PPG  requires  the  solicitation  to 
state  [Ref.  2:p.  6]: 

1.  The  government  will  conduct  a  performamce  risk 
assessment  based  on  the  past  performamce  of  the  offerors  and 
their  proposed  subcontractors  as  it  relates  to  the 
probcd}ility  of  successful  accomplishment  of  the  work 
required  by  the  solicitation; 

2.  In  conducting  the  performamce  risk  assessment,  the 
government  may  use  data  provided  by  the  offeror  and  data 
obtained  from  other  sources; 

3.  While  the  Government  may  elect  to  consider  data  obtained 
from  other  sources,  the  burden  of  providing  complete  past 
performamce  information  rests  with  the  offeror. 


When  past  performance  is  to  be  assessed,  it  may  be 
necessary  to  explain  the  methodology  to  the  offerors  during 
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the  pre- solicitation  and  pre-proposal  conferences. 
Additionally,  Section  L  of  the  solicitation  should  instruct 
offerors  to  disclose  information  concerning  contracts  or 
subcontracts  for  which  similar  work  has  been  performed.  It  is 
importamt  for  prime  contractors  to  describe  their 
subcontracting  plaui  and  the  ejected  work  to  be  performed  so 
an  assessment  of  potential  subcontractors  can  also  be  made. 
[Ref.  2:p.  6] 

6.  Sources  of  Data 

The  Army  Material  Command  established  the  Contractor 
Information  System  (CIS)  to  provide  sources  of  past 
performcuice  information.  The  CIS  does  not  contain  contractor 
report  cards .  It  contains  only  factual  information  pertaining 
to  a  contractor.  It  is  an  electronic  telephone  book.  [Ref. 
2:p.  7] 

The  PRAG  can  obtain  data  from  AMC  or  the  Defense 
Logistics  Agency  (DLA) .  However,  the  major  sources  of 
information  are  the  references  cited  in  the  contractor's 
proposal.  The  PRAG  determines  past  efforts  that  may  be 
relevant  to  the  current  contract.  Previous  contracts  can  be 
relevant  in  whole  or  relevcuit  in  only  a  few  specific  areas. 
An  evaluation  of  past  performauice  is  more  vicd)le  if  a  previous 
contract  has  just  recently  been  completed  or  is  still  in 
progress.  Three  years  is  a  normal  time- frame  that  has  been  a 
successful  cutoff  for  most  activities.  [Ref.  2:pp.  6,  7] 
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OPPP  Policy  Letter  92-5  requires  activities  not  only 
to  evaluate  past  performance  during  the  source  selection 
process,  but  also  requires  a  contractor  performeuice  evaluation 
at  the  completion  of  all  contracts  in  excess  of  $100,000.  The 
evaluations  are  placed  in  the  contract  file  and  caui  be  used  by 
the  PRAG  at  a  later  date  for  future  assessments.  [Ref.  2:p.  8] 

Commercial  references  Ceui  also  be  used  by  the  PRAG, 
however  Government  sources  are  preferred  when  they  are 
availcUsle.  Information  from  commercial  euid  foreign  government 
sources  should  be  verified  prior  to  being  used  for  an 
evaluation.  It  is  not  necessary  to  require  similar  references 
from  the  other  offerors  as  long  as  sufficient  data  are 
availeible.  [Ref.  2:p.  8] 

7.  Gathering  Data 

The  PRAG  primarily  gathers  information  through 
questionnaires  and  telephone  interviews.  Experience  has  shown 
that  the  best  method  is  to  initially  send  out  questionnaires 
and  to  follow  up  promising  sources  with  a  telephone  interview. 
[Ref.  2:p.  8] 

Efforts  for  gathering  information  are  often  done  on  eui 
individual  basis.  However,  it  was  recommended  that  the  PRAG 
assemble  a  group  to  conduct  interviews.  This  offers  the 
ed)ility  of  getting  instant  feedback  from  one  auiother.  For 
each  previous  contract  reviewed,  the  PRAG  should  interview  at 
least  two  references.  [Ref.  2:p.  9] 
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Prior  to  an  Interview  or  sending  out  a  questionnaire 
several  questions  should  be  prepared.  The  interviewer  should 
inform  the  reference  that  their  conversation  will  be  followed 
up  with  a  memorandum  to  verify  the  accuracy  of  the  information 
discussed.  Care  must  be  taken  to  ensure  the  accuracy, 
clarity,  emd  legibility  of  the  summaries  which  may  be  the  only 
back-up  supporting  the  PRAG's  recommendation.  [Ref.  2:p.  9] 

8.  Assigning  Performance  Risk  Ratings 

When  assigning  a  performauice  risk  rating,  the  PRA6 
must  consider  the  number  euid  severity  of  problems,  corrective 
actions  taken,  and  the  contractor's  overall  work  record.  If 
past  performance  problems  exist,  the  PRAG  must  determine  the 
role  the  Government  may  have  played  in  their  occurrences. 
[Ref.  2:p.  11] 

The  PRAG  assessment  is  often  subjective  in  nature.  It 
is  difficult  to  develop  a  mathematical  model  from  which  a 
score  is  determined.  The  assessment  should  include  a 
description  supporting  the  assigned  performance  category. 
PRAG  members  must  concern  themselves  with  promises  made  by 
offerors  to  correct  past  failures.  A  promise  does  not  improve 
performance,  however  demonstrated  corrective  actions  can  lower 
the  risk  of  future  performance  failures.  [Ref.  2:p.  11] 

9.  The  Assessment  Report 

The  report  developed  by  the  PRAG  should  include  the 
information  required  by  the  source  selection  authority  to  make 


15 


an  Informed  judgment.  Factual  data  giving  specific  examples 
of  performance  failures  should  be  included  to  support  the 
conclusions.  The  PRAG  should  review  the  report  to  ensure  it 
is  accurate .  When  a  luicuiimous  risk  assessment  ceumot  be 
reached,  the  PRAG  should  include  a  dissenting  opinion  as  part 
of  the  report.  [Ref.  2:  p.ll] 

C.  NAVY'S  RBD,  YELLOW,  OSBBN  PROGRAM 
1 .  Backgrotind 

The  Red,  Yellow,  Green  (RY6)  Program  was  developed 
from  the  Product  Deficiency  Reporting  and  Evaluation  Program 
(PDREP) .  It  is  a  tool  that  allows  the  use  of  a  contractor's 
product  quality  history  to  help  reduce  the  risk  of  receiving 
non- conforming  goods.  [Ref.  16 :p.  1] 

The  evaluation  part  of  the  PDREP  is  the  Contractor 
Evaluation  System  (CES) .  Contractor  history  is  available  on¬ 
line  over  dial-up  telephone  lines  from  the  Naval  Material 
Quality  Assessment  Office.  [Ref.  16 :p.  1]  An  interview 
conducted  with  Mr.  Kaul,  ASO,  revealed  that  problems  currently 
exist  in  retrieving  data  from  the  CES.  With  one  hard  line 
going  to  the  system,  it  is  difficult  to  access  the  data  base. 
[Ref.  27] 

RY6  uses  the  historical  performance  of  a  contractor  to 
determine  the  degree  of  risk  of  receiving  a  non- conforming 
product .  RYG  procedures  can  be  used  in  two  ways :  the 
Technical  Evaluation  Adjustment  (TEA)  method  and  the  Greatest 
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Value/Best  Buy  (GV/BB)  method.  A  color  (Red/ Yellow/or  Green) 
Is  assigned  to  a  contractor  for  each  Federal  Supply 
Classification  (FSC) .  [Ref.  16 :p.  1] 

A  TEA  is  the  anticipated  costs  required  by  the 
Government  for  quality  actions  to  reduce  the  risk  of  receiving 
a  non- conforming  product  from  a  red  or  yellow  offeror.  TBAs 
are  added  to  an  offeror's  evaluation  to  determine  the  best 
overall  value  to  the  Government.  [Ref.  16 :p.  1-2]  Additional 
quality  assurance  actions  include:  Pre-Award  Surveys,  Post- 
Award  Orientation  Conferences,  Government  Source  Inspections, 
Receipt  Inspections  at  the  Source,  Receipt  Inspections  at  the 
Destination  and  Quality  Assurance  Letters  of  Instruction. 
Additional  actions  may  be  required  depending  on  the 
contractor's  expected  performance.  [Ref.  16: Appendix  G] 

Under  the  RYG  GV/BB  method,  an  activity  will  include 
both  price  and  the  RYG  classification  as  evaluation  factors  in 
the  source  selection  plan.  Consideration  can  be  given  to 
technical  factors  depending  on  the  requirements  of  the 
procurement.  This  method  is  useful  for  large  procurements 
where  the  adjustment  of  TEAS  would  not  displace  offerors. 
[Ref.  16 :p.  2] 

Monthly,  a  CES  program  produces  color  classifications 
for  each  FSC  that  a  contractor  has  product  quality  history 
(deficiency  reports,  surveys,  etc.).  The  predefined  criteria 
listed  in  Appendix  A  of  this  thesis  are  used  in  determining 
the  color  classification.  [Ref.  16 ;p.  2] 


17 


The  RY6  progrcun  consists  of  three  procurement 
procedures:  Slit^lified  Small  Purchase,  Major  Purchase  etnd 
Greatest  Value/Best  Buy  Source  Selection  emd  Evaluation. 
Activities  select  the  commodities  for  which  they  desire  to  use 
RYG  procedures.  Once  these  commodities  are  selected,  all 
subsequent  purchases  must  be  made  under  RYG  procedures  unless 
a  waiver  is  received  from  the  Chief  of  the  Contracting  Office. 
RYG  does  not  eliminate  the  requirement  for  a  contracting 
officer  to  make  a  responsibility  determination.  The  FSC 
classification  alone  Is  not  sufficient  for  determining 
responsibility,  nor  does  It  prevent  a  contractor  from 
con5)eting  on  a  contract.  [Ref.  16 :p.  3] 

2 ,  Red/Yel low/Green  Procedures 

a.  Sinplifled  Small  Purcbaae  Procedures 

RYG  Simplified  Small  Purchase  Procedures  apply  to 
solicitations  when  sln^llfled  small  purchases  are  used. 
During  evaluation,  the  applicable  TEA  Is  added  to  a  supplier's 
quoted  price.  After  consideration  of  other  price -related 
factors,  this  becomes  the  basis  for  determining  award  of  the 
purchase  order.  [Ref.  16 :p.  5] 

When  synopsizing  the  requirement  in  the  Commerce 
Business  Daily,  the  synopsis  must  state  that  the  award  will  be 
based  on  factors  other  than  price,  including  historical 
quality  performance  data.  Additionally,  the  written  quotation 
request  shall  include  the  required  clauses  to  advise  the 
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contractor  of  RYG  procedures.  For  oral  requests,  the  clauses 
shall  be  discussed  with  the  supplier.  [Ref.  16 :p.  5] 

Color  classifications  under  simplified  ssiall 
purchase  procedures  are  defined  In  the  following  manner  [Ref 

16:p.  6] : 

Green  -  a  green  classification  meouis  the  contractor  has 
demonstrated  satisfactory  performance  In  delivering  a 
quality  product  or  has  demonstrated  satisfactory 
capability  to  deliver  a  quality  product,  for  a  specific 
FSC.  Evaluation  of  quotations  for  award  may  continue  In 
accordance  with  estcd>llshed  acquisition  regulations 
without  consideration  of  unusual  quality  actions.  Factors 
other  than  quality  determine  the  offeror's  eligibility  for 
award . 

Yellow  -  A  yellow  classification  means  that  the  contractor 
has  sufficient  negative  quality  history  for  a  specific  FSC 
to  require  additional  quality  assurcuice  actions  aimed  at 
reducing  the  risk  of  poor  quality  products  being  delivered 
to  the  Navy.  A  TEA,  representing  the  cost  to  the 
Government  of  doing  additional  quality  assurance  actions 
to  reduce  the  risk  of  receiving  poor  quality  products.  Is 
added  to  the  quoted  price  of  the  offemr. 

Red  -  a  red  classification  means  that  the  contractor's 
negative  quality  history  for  a  specific  FSC  is  serious 
enough  to  rec[ulre  additional  quality  assurcmce  actions 
aimed  at  reducing  the  risk  of  poor  quality  products  being 
delivered  to  the  Navy.  A  TEA,  representing  the  costs  to 
the  Government  of  doing  additional  quality  assuremce 
actions  to  reduce  the  risk  of  receiving  poor  equality 
products.  Is  added  to  the  quoted  price  of  the  offeror.  In 
addition,  the  head  of  the  contracting  office  must  approve 
the  award. 

Insufficient  Data  -  First  time  offerors  or  offerors  for 
whom  current,  up-to-date  quality  performance  history  is 
unavailable  are  labeled  "Insufficient  data"  contractors. 
Additional  quality  assurance  actions  may  be  needed,  but 
TEAS  will  not  be  assessed  for  "Insufficient  data" 
contractors . 
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b.  Major  Purchase  Procedures 


The  RYG  Major  Purchase  Procedures  are  used  for 
negotiated  procurements  when  Simplified  Small  Purchase 
Procedures  or  RYG  Greatest  Value/Best  Buy  Procedures  are  not 
used  [Ref  16 :p.  7]. 

While  evaluating  proposals,  the  cognlzeuit  (juality 

staff  will  review  the  RYG  classification  database  auid 

recommend  to  the  Contracting  Officer  the  necessary  quality 

assurance  actions  so  a  TEA  amount  can  be  determined.  FSCs 

with  the  same  color  classification  are  evaluated  individually. 

Based  on  historical  performance  of  the  contractor,  different 

quality  assurance  actions  may  be  required  even  if  they  have 

the  same  color  classification.  After  evaluating  other  cost 

related  factors  and  adding  in  the  TEAs,  this  becomes  the  basis 

for  awarding  the  contract.  [Ref.  16 :p.  7] 

When  synopsizing  the  requirement  in  the  Commerce 

Business  Daily,  the  synopsis  must  state  that  the  award  will  be 

based  on  factors  other  than  price,  including  historical 

quality  performance  data.  Additionally,  the  written  proposal 

request  shall  include  the  required  clauses  to  advise  the 

contractor  of  RYG  procedures.  For  oral  requests,  the  clauses 

shall  be  discussed  with  the  supplier.  [Ref.  16 :p.  5] 

The  solicitation  shall  state  [Ref.  16:p.  10]; 

This  procurement  is  part  of  the  Navy's  Contractor 
Evaluation  System,  Red/Yellow/Green  Program.  Award  will 
be  based  on  the  Contracting  Officer's  decision  as  to  which 
offer  provides  the  best  value  to  the  Navy  -  price,  past 
quality  performance,  and  other  factors  considered. 
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Details  are  provided  in  provisions  entitled  'NOTICE  TO 
PROSPECTIVE  OFFERORS'  (Section  L)  and  'ADDITIONAL  FACTORS 
-  CONTRACTOR  EVALUATION  SYSTEM,  RED/YELLOW/GREEN  PROGRAM' 
(Section  M) . 

Color  classifications  under  Major  Purchase 

Procedures  are  defined  in  the  following  mcuiner  [Ref  16. p.  8]  : 

Green  -  a  green  classification  means  the  contractor  has 
demonstrated  satisfactory  performance  in  delivering  a 
quality  product  or  has  demonstrated  satisfactory 
capability  to  deliver  a  quality  product,  for  a  specific 
FSC.  Evaluation  of  quotations  for  a%fard  may  continue  in 
accordance  with  established  acquisition  regulations 
without  consideration  of  unusual  quality  actions.  Factors 
other  than  quality  determine  the  offeror's  eligibility  for 
award . 

Yellow  -  A  yellow  classification  means  that  the  contractor 
has  sufficient  negative  quality  history  for  a  specific  FSC 
to  require  additional  quality  assurance  actions  aimed  at 
reducing  the  risk  of  poor  quality  products  being  delivered 
to  the  Navy.  Cognizant  quality  personnel  shall  review 
both  the  procurement  package  and  the  offeror's  performance 
history  in  the  RYG  datadaase  to  determine  the  proper 
quality  control  measures.  The  applicable  TEA  is  added  to 
the  quoted  price  to  determine  the  actual  cost  to  the 
Government  of  performing  additional  quality  assurcuice 
actions  to  reduce  the  risk  of  receiving  poor  quality 
products . 

Red  -  a  red  classification  means  that  the  contractor's 
negative  quality  history  for  a  specific  FSC  is  serious 
enough  to  require  additional  quality  assurance  actions 
aimed  at  reducing  the  risk  of  poor  quality  products  being 
delivered  to  the  Navy.  In  addition  to  forwarding  the 
procurement  package  to  the  cognizant  Quality  staff  for 
review  and  determination  of  quality  assurance  actions,  the 
award  must  be  approved  by  the  head  of  the  contracting 
office. 

Insufficient  Data  -  First  time  offerors  or  offerors  for 
whom  current,  up-to-date  quality  performance  history  is 
unavailable  are  labeled  "insufficient  data"  contractors. 
Additional  quality  assurance  actions  may  be  needed,  but 
TEAS  will  not  be  assessed  for  "insufficient  data" 
contractors . 
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c.  Greatest  VgLLue/Beat  Buy  ProcedwcBB 

RYG  Greatest  Value/Best  Buy  Procedures  shall  be 
considered  when  "sealed  bidding"  procedures  would  otherwise  be 
appropriate.  It  applies  to  negotiated,  competitive 
solicitations  when  other  than  simplified  small  purchase 
procedures  or  major  purchase  with  TEAS  are  used.  [Ref.  16 :p. 
91 


Under  the  RYG  GV/BB  Procedures,  source  selection 
criteria  shall  Include  price  euid  RYG  classifications  as  the 
evaluation  factors  unless  prior  approval  Is  given  from  the 
Chief  of  the  Contracting  Office.  Activities  may  select  the 
source  selection/evaluation  methodology  best  suited  to  meet 
the  needs  of  the  requirement.  The  quality  staff  shall  prepare 
the  source  selection  plan  including  the  criteria  for 
evaluating  red,  yellow  and  green  FSCs.  [Ref.  16 :p.  9] 

When  RYG  GV/BB  Procedures  are  used,  the  synopsis 
In  the  CBD  must  state  that  award  will  be  based  on  price  eUid 
historical  quality  performance.  For  purchases  less  than 
$25,000,  that  do  not  require  synopsis,  offerors  will  be 
advised  of  the  RYG  procedures.  The  solicitation  shall  state 
[Ref.  16:p.  10] ; 

This  procurement  is  part  of  the  Navy's  Contractor 
Evaluation  System,  Red/Yellow/Green  Program.  Award  will 
be  based  on  the  Contracting  Officer's  decision  as  to  which 
offer  provides  the  best  value  to  the  Navy  -  price,  past 
quality  performance,  and  other  factors  considered. 
Details  are  provided  in  provisions  entitled  'NOTICE  TO 
PROSPECTIVE  OFFERORS'  (Section  L)  and  'ADDITIONAL  FACTORS 
-  CONTRACTOR  EVALUATION  SYSTEM,  RED/YELLOW/GREEN  PROGRAM' 
(Section  M) . 
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The  Contracting  Officer,  after  receiving 
proposals,  shall  forward  the  names  of  offerors  to  the  Quality 
staff.  The  Quality  staff  forwards  em  evaluation  that  Includes 
the  color  classification  of  cm  offeror's  commodity.  A 
narrative  e3q)lalnlng  the  relative  differences  between  offerors 
shall  be  provided  when  commodities  of  different  offerors  have 
the  same  color  classification.  Prior  to  contract  award,  the 
Quality  staff  shall  determine  the  need  for  quality  assurauice 
actions  to  include  as  part  of  the  contract.  [Ref.  16 :p.  9] 

The  following  are  provided  as  a  guide  for 
establishing  source  selection  criteria  euid  est6d>llshlng 
appropriate  evaluation  measurements  for  source  selection  [Ref. 

16;pp.  10-11] ; 

Green  -  a  green  classification  meeuis  the  contractor  has 
demonstrated  satisfactory  performauace  In  delivering  a 
quality  product  or  has  demonstrated  satisfactory 
capaUDlllty  to  deliver  a  quality  product,  for  a  specific 
FSC.  Commodities  with  this  classification  would  normally 
receive  an  adjectival  rating  of  "good"  to  "excellent," 
based  on  the  criteria  In  i^^pendlx  A. 

Yellow  -  a  yellow  classification  means  that  the  contractor 
has  sufficient  negative  quality  history  for  a  specific  FSC 
to  require  additional  quality  assuremce  actions  aimed  at 
reducing  the  risk  of  poor  quality  products  being  delivered 
to  the  Navy.  Commodities  with  this  classification  could 
receive  an  adjectival  rating  of  "average,  "  "good,  "  or  even 
"marginal"  --  depending  on  the  number  of  quality  problems 
euid  the  severity  of  each. 

Red  -  a  red  classification  means  that  the  contractor's 
negative  quality  history  for  a  specific  FSC  Is  serious 
enough  to  require  approval  by  the  head  of  the  contracting 
activity  to  award.  Previous  experience  with  the 
contractor's  commodity  has  yielded  products  that  were  less 
them  satisfactory  or  totally  unacceptedale .  The  normal 
adjectival  rating  would  be  "marginal"  or  "unacceptcUsle. " 
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Insufficient  Data  -  First  time  offerors  or  offerors  for 
whom  current,  up-to-date  quality  performance  history  is 
unavailable  are  labeled  "insufficient  data"  contractors. 

An  "insufficient  data"  commodity  shall  be  evaluated  solely 
on  the  basis  of  "lowest  price,  technically  acceptad>le . " 
Activities  are  encouraged  to  request  past  quality 
performance  data  in  the  solicitation  for  consideration  in 
the  responsibility  determination  process. 

D.  AIR  FORCB  PROCBDURBS 

The  Air  Force  utilizes  two  initiatives  for  meocimizlng  its 
use  of  past  performance  as  an  evaluation  for  source  selection: 
the  Contractor  Performance  Assessment  Reporting  System  (CPARS) 
euid  the  Performance  Risk  Assessment  Group  (PRAG) .  The  CPARS 
is  used  to  both  record  and  disseminate  information  concerning 
the  performance  of  a  contractor.  The  PRAG  is  a  pcinel  of 
acquisition  professionals  that  evaluate  the  degree  of 
performance  risk  of  a  contractor's  proposal  for  each  source 
selection.  [Ref.  23:pp.  1-2] 

1.  Contractor  Perfomance  Assessment  Reporting  System 
The  CPARS  is  an  initiative  used  by  the  Air  Force. 
Each  progreun  manager  in  the  Air  Force  Material  Command  is 
recjuired  to  prepare  an  annual  assessment  of  a  contractor's 
performance.  This  evaluation  is  based  on  the  following  ten 
criteria:  product /system  performance,  software  development, 
engineering  design  support,  schedule,  cost  performance, 
product  assurance,  test  cuid  evaluation,  management 
responsiveness,  integrated  logistics  support,  and  siibcontract 
management.  The  program  manager  uses  a  four  level  scale 
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ranging  fron  unsatisfactory  to  exceptional.  His  determination 
must  be  based  on  objective  results  that  are  Included  In  the 
narrative  of  the  assessment  report.  [Ref  23 :p.  1-3] 

The  reporting  syst^  begins  with  the  program  memager 
soliciting  functional  experts  for  objective  results  In  their 
areas.  A  preliminary  report  Is  developed  amd  forwarded  to  the 
contractor  for  review.  The  contractor  has  30  days  In  which  to 
submit  a  response.  The  response  Is  optional  amd  should 
concentrate  on  the  objective  data  In  the  report.  [Ref.  23 :p. 
1-31 


Once  the  program  mamager  receives  the  report  from  the 
contractor,  he  cam  revise  the  report  on  a  new  form,  (attaching 
the  old  report) ,  or  madce  no  chamges.  Once  completed,  he  signs 
the  report  atnd  sends  It  to  the  next  higher  level  In  the 
commamd.  The  reviewing  official  ensures  the  report  Is 
consistent  with  other  evaluations  of  the  contractor  for  which 
the  evaluation  Is  based.  Copies  of  the  report  are  then  sent 
to  each  product  division.  [Ref.  23:  pp.  1-3,  1-4]  These 
reports  then  become  avalladjle  for  use  by  the  PRAG  when 
developing  an  assessment  of  a  contractor's  performance  [Ref. 
23:p.  1-6].  According  to  Air  Force  regulation  [Ref  23:pp.  1- 
4,  1-5] : 

The  sole  purpose  of  the  CPARS  Is  to  provide  prograun 
mamagement  Input  for  a  commaind-wlde  performance  data  base 

used  In  -  source  selection.  Performance  assessment 

will  be  used  as  an  aid  In  awarding  contracts  to 
contractors  that  consistently  produce  quality  products 
that  conform  to  requirements  within  contract  schedule  and 
costs . 
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The  CPARS  does  have  same  limited  use.  For  contractors 
that  have  not  been  awarded  a  contract  since  It  began,  no 
performance  data  will  exist  [Ref.  23 :p.  1-5].  In  these 

Instemces,  the  PRA6  can  rely  on  questionnaires  to  program 
Qicmagers  for  data  specific  to  a  given  contractor.  In  the  long 
run  however,  the  CPARS  will  be  the  primary  source  of  data 
avallcQsle  for  use  by  the  PRAG.  [Ref.  23:pp.  1-6] 

2.  Performance  Risk  Assessment  Orovp 

The  Information  from  the  CPARS  becomes  very  valuable 
to  the  PerfomiflUice  Risk  Assessment  Group  responsible  for 
providing  a  performauice  evaluation  and  risk  assessment  of  a 
contractor.  Their  goal  Is  to  determine  a  contractor's  cd>lllty 
to  perform  on  a  proposed  contract  given  their  previous 
performance  record.  The  assessment  considers  previous 
problems  encoxmtered  by  the  contractor,  the  steps  taken  to 
correct  deficiencies,  euid  the  contractor's  trends  of 
performance.  The  results  of  the  PRAG  are  not  Intended  to 
equate  to  a  numerical  score  by  which  the  contractor  Is  either 
a  "go”  or  a  "no-go”,  but  rather  Is  an  excunlnatlon  of  the 
avalled)le  performance  data  that  are  relevemt  to  the  source 
selection.  [Ref.  23 :p.  1-5] 

The  PRAG  not  only  relies  on  data  from  the  CPARS,  which 
are  categorized  as  contract  specific,  but  also  utilizes 
contractor -plant  Information  and  a  contractor's  self- 
assessment.  Contractor  pleuit  Information  Is  provided  by  the 
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Defense  Contract  Management  Command  (DCMC) .  Information 
compiled  by  these  sources  are  assessments  of  the  internal 
operations  and  nsmagement  functions  of  am  orgamization.  The 
contractor's  self-assessment  is  provided  in  the  response  to 
the  request  for  proposal.  The  proposal  indicates  the  level  of 
experience  that  the  contractor  has  with  the  type  of  %K>rk 
required  for  the  successful  completion  of  the  contract.  [Ref. 
23;pp.  1-6,  1-7] 

B.  SBJII.BD  BIDDING 

Sealed  bidding  is  one  of  the  two  basic  procurement  methods 
used  by  the  Government.  The  Con^etltlon  in  Contracting  Act 
requires  procuring  agencies  to  use  competition  in  the 
selection  of  contractors  to  supply  goods  or  services  [Ref. 
3:p.  2-17] .  Under  sealed  bidding,  ccmpetition  is  achieved  by 
evaluating  price.  Offerors  submit  bids  in  response  to  a 
solicitation,  the  bids  are  publicly  opened  at  a  designated 
time  and  location,  emd  the  contract  is  awarded  to  the 
responsive  emd  responsible  offeror  who  submitted  the  lowest 
bid. 

Several  circumstances  must  exist  for  sealed  bidding  to  be 
used.  First,  there  must  be  enough  time  avalled>le  for  the 
solicitation,  submission,  smd  evaluation  of  sealed  bids. 
Second,  award  must  be  based  on  price  and  price- related 
factors.  Third,  discussions  Ceumot  be  held  with  contractors 
concerning  their  bids  and  last,  there  is  an  e3q)ectation  that 
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more  than  one  bid  will  be  received.  If  an  acquisition  cannot 
meet  these  criteria,  then  sealed  bidding  cannot  be  used.  [Ref. 
3:p.  3-2] 

Under  sealed  bidding  procedures,  two  areas  must  be 
addressed:  the  responsiveness  of  bids  amd  the  responsibility 
of  bidders.  A  responsive  bid  is  am  unqualified  offer  that 
meets  the  requirements  as  specified  in  the  Invitation  for  Bid 
(IFB) .  However,  to  be  eligible  for  award,  the  bidder  must 
also  be  responsible.  Determination  of  whether  a  bid  is 
responsive  is  made  immediately  after  the  opening  of  bids.  A 
determination  of  responsibility  is  made  as  soon  as  possible, 
before  contract  award.  [Ref.  3:p.  3-23] 

F.  COMPBTZTZVB  NEGOTIATIONS 

Con^etitive  negotiations  is  the  other  basic  procurement 
method  which  allows  the  Government  to  award  contracts  on 
criteria  other  than  price  and  price  related  factors.  This 
method  of  contracting  permits  the  greatest  use  of  past 
perfomiemce  in  the  source  selection  process.  Under 
con^etitive  negotiations,  the  contracting  officer  is  free  to 
consider:  (1)  the  offeror's  e3q>erience,  (2)  technical  and 
management  capcUsility,  (3)  cost  information,  and  (4)  a 
contract  type  which  the  offeror  is  willing  to  accept.  [Ref.  3: 
p.  4-2] 

The  request  for  proposal  (RFP)  is  used  under  con?)etitive 
negotiations  procedures.  The  RFP  emd  RFQ  (Request  for 
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Quotation)  are  used  "to  cc»!Bnunlcate  Government  requirements  to 
prospective  contractors  amd  to  solicit  proposals  or  quotations 
from  them.  ”  The  distinction  between  RFPs  emd  RFQs  has  become 
somewhat  blurred  over  the  last  several  years;  the  terms  are 
sometimes  used  interchcuigeably,  especially  at  the  subcontract 
level.  [Ref.  3;pp.  4-3,  4-4] 

Multiple  evaluation  factors  are  permitted  in  con^etitive 
negotiations.  To  achieve  optimal  competition  it  is  necessary 
for  the  Government  to  disclose  to  the  offerors  the  factors 
that  will  be  evaluated  euid  the  relative  order  of  in^ortcuice  of 
these  factors.  There  is  generally  no  groimds  for  questioning 
these  factors  as  long  as  they  are  set  forth  in  the  RFP  auid 
relate  to  the  purpose  of  the  procurement.  The  factors  used 
will  depend  on  the  circumstances  of  the  procurement.  [Ref. 
3:p.  4-5]  Past  performance  can  be  evaluated  as  an  evaluation 
criterion  euid  has  the  seune  disclosure  requirements  as  the 
other  factors. 

The  evaluation  factors  and  their  relative  weights  as 
disclosed  in  the  RFP  must  be  used  in  the  evaluation.  All 
offerors  must  be  informed  of  any  changes  in  the  evaluation 
factors  or  changes  in  their  relative  weights.  It  is  not 
necessary  to  disclose  the  actual  numerical  weights  of  the 
evaluation  criteria.  [Ref.  3;p.  4-6] 

After  proposals  are  submitted,  oral  or  written  discussions 
are  normally  held  with  offerors  that  are  in  the  competitive 
range.  At  the  conclusion  of  discussions,  offerors  submit 
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their  best  auid  final  offers  (BAFOs) .  Once  the  final  round  of 
BAFOs  are  submitted,  a  winner  is  determined.  [Ref.  3:pp.  4>8, 
4-9] 

G.  RBSP0NSZBILZT7 

The  responsibility  deteimination  of  a  contractor  is 
inportemt  since  price  alone  does  not  necessarily  guarantee  the 
delivery  of  conforming  goods.  Default,  late  deliveries,  or 
other  measures  of  non-conformance  can  result  in  increased 
contractual  and  administrative  costs  not  originally 
anticipated.  No  contract  award  can  be  made  unless  the 
contracting  officer  has  made  a  positive  assessment  of  a 
contractor's  responsibility.  If  enough  information  is  not 
availeOsle  to  clearly  make  a  responsibility  determination,  the 
contractor  will  be  deemed  non- responsible.  [Ref.  10: Subpart 
9.103]  Additional  requirements  exists  for  small  businesses  as 
described  in  the  FAR  [Ref.  10: Subpart  19.6]. 

The  FAR  describes  several  standards  that  must  be  evaluated 
to  determine  the  overall  responsibility  of  a  prospective 
contractor.  The  contractor  must  [Ref.  10: Subpart  9.104-1]: 

(1)  Have  adequate  financial  resources  to  perform  the 
contract  or  the  cibility  to  obtain  them; 

(2)  Be  cUole  to  con^ly  with  the  required  or  proposed 
delivery  or  performance  schedule,  taking  into  consideration 
all  existing  commercial  and  Government  business  commitments; 

(3)  Have  a  satisfactory  performance  record; 
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(4)  Have  a  satlsfactozy  record  of  Integrity  and  business 
ethics ; 

(5)  Have  the  necessary  orgainization,  experience,  accounting 
and  operational  controls  and  technical  skills,  or  the 
ability  to  obtain  them  (including,  as  appropriate,  such 
elements  as  production  control  proc^ures,  property  control 
systems,  quality  assurance  measures,  cuid  safety  programs 
applicable  to  materials  to  be  produced  or  services  to  be 
performed  by  the  prospective  contractor  euid  subcontractors.) 

(6)  Have  the  necessary  production,  construction  and 
technical  equipment  euad  facilities,  or  the  ability  to  obtain 
them;  and 

(7)  Be  otherwise  qualified  eind  eligible  to  receive  an  award 
under  applicad^le  laws  and  regulations. 


Special  standards  of  responsibility  may  also  be  required  for 
an  acquisition  that  in  previous  experience  has  demonstrated 
specialized  experience  or  facilities  to  perform  contractual 
requirements.  In  these  circumstcinces,  the  solicitation  will 
define  all  special  standards  that  must  be  met.  [Ref. 
10: Subpart  9.104-2] 

As  stated  in  (3)  cibove,  a  contractor's  past  performance 
record  must  be  evaluated  to  make  a  responsibility 
determination.  A  contractor  that  has  recently  exhibited 
serious  deficiencies  in  contract  performance  shall  be 
considered  non- responsible,  unless  the  contracting  officer 
makes  a  determination  that  the  performance  was  beyond  the 
contractor's  control  and  that  corrective  action  has  been  taken 
to  prevent  similar  occurrences  in  the  future.  The  failure  of 
a  contractor  to  apply  all  his  efforts  to  cortplete  a  contract 
is  evidence  of  non- responsibility.  The  inability  of  a 
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contractor  to  meet  quality  requirements  is  also  a  factor  to 
consider.  In  medcing  a  responsibility  determination,  the 
contracting  officer  must  consider  the  number  of  contracts  for 
which  the  contractor  is  involved  and  the  extent  of  non¬ 
conformance  in  each  contract.  If  the  pending  contract 
requires  a  siibcontractor  plan,  the  contracting  officer  must 
evaluate  the  contractor's  edjility  in  developing  and  complying 
with  such  a  plan.  [Ref.  10: Subpart  9.104-3] 

Generally,  prime  contractors  are  required  to  determine  the 
responsibility  of  subcontractors.  A  contract  may  require  the 
prime  to  provide  written  evidence  of  a  subcontractor' s 
responsibi'' ■’ t  '.  However,  when  the  subcontract  involves  cm 
v'rgent  rec^j  ■  cement  or  substantial  subcontracting,  the 
contracting  officer  may  make  a  responsibility  determixiation. 
These  circximstances  will  require  the  same  conformance  to 
standards  as  rec[uired  of  the  prime  contractor.  [Ref. 
10:Sxibpart  9.104-4] 

H.  SUMMARY 

This  chapter  has  introduced  the  Army's  PRAG  Program,  the 
Navy's  RYG  Progrcun  and  the  Air  Force's  CPARS.  The  Navy  and 
Air  Force  Programs  have  been  introduced  to  provide  an  insight 
to  the  procedures  used  by  the  other  Services  that  could  be 
included  as  part  of  the  Army's  PRAG  Program.  These  data  will 
be  used  in  the  analysis  of  the  PPG  in  Chapter  IV. 
Additionally,  a  summary  of  competitive  negotiations,  sealed 
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were 


bidding  and  the  determination  of  responsibility 
introduced  to  correlate  the  use  of  past  performamce  in  these 
procedures . 
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III.  ZMDUST&T  PROCBDUBBS 


A.  DITRQDUCTZON 

This  chapter  presents  the  data  amd  analysis  of  the  results 
of  interviews  conducted  with  commercial  organizations. 
Questions  focused  on  the  procedures  used  by  industry,  the 
evaluation  factors  and  the  recency  amd  relevauicy  of  data. 
Questions  were  selected  so  a  correlation  could  be  better  made 
between  industry  results  and  the  Army's  PPG. 

B.  PROCEDURES  USED  BY  COMMERCIAL  ORGBHIZATIONS 
1.  Vought  Aircraft!  Dallas,  TZ 

a.  Background 

The  prograuns  at  Vought  Aircraft  Con^amy  include 
the  B-2  and  the  C-17.  They  manufacture  components  for  the  B-2 
advanced  technology  bomber;  C-17  engine  nacelles  and  vertical 
and  horizontal  stad)ilizers  for  McDonnell  Douglas  and  Boeing. 
[Ref.  l:p.  26]  Mr.  John  Summit,  the  Mauiager;  Composites, 
Tooling,  Assemblies  and  Outside  Processing  Procurement,  was 
interviewed  to  develop  an  understanding  of  procedures  used  at 
Vought  Aircraft. 

b.  Assessing  Past  Performance 

The  methodology  used  by  Vought  Aircraft  consists 
of  the  SRS  (Supplier  Rating  System)  and  the  SPIP  (Supplier 
Performance  Indicator  Prograun)  .  Past  performance  is  am 
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Integral  part  of  their  source  selection  process  and  Is  key  In 
a  number  of  a  system's  Items.  [Ref.  21] 

(1)  Supplier  Rating  System,  The  SRS  looks  at  cost 
and  on  time  delivery  Including  appropriate  paperwork  and 
certification  documents.  A  supplier's  performance  Is  not 
evaluated  alone,  rather  It  Is  conqpared  to  the  performance  of 
the  supplier's  peers.  [Ref.  21] 

(2)  Supplier  Performance  Indicator  Program.  The 
SPIP  Includes  quality  which  affects  the  cost  of  doing  business 
with  a  supplier.  In  evaluating  quality  one  must  look  at  the 
commodity.  Historically,  some  ctmimodltles  may  be  prone  to 
certain  problems  and  defects.  With  the  purchase  of  rivets  for 
exan^le,  allowances  could  be  made  for  a  rejection  rate  of  10% 
out  of  5000  pieces.  In  this  Instance  It  would  be  more  costly 
to  develop  a  process  for  reducing  the  rejection  rates  versus 
throwing  out  the  bad  rivets.  So,  a  90%  quality  rating  would 
be  acceptcd)le.  With  a  skin  commodity  though,  like  part  of  an 
aircraft  wing,  one  may  require  a  97%  rating.  In  this  Instance 
It  Is  not  practical  to  throw  out  the  bad  lots.  [Ref.  21] 

The  SPIP  Includes  system  rework  costs; 
the  extra  costs  the  con^any  could  expect  to  Incur  If  a 
supplier  Is  selected.  In  comparing  two  suppliers.  It  may  be 
cheaper  In  the  long  run  to  buy  a  more  expensive  part  If  It 
requires  less  rework.  Vought  Aircraft  Is  looking  for  the  best 
value.  Sometimes  there  Is  a  known  monetary  value  a  proposal 
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can  be  adjusted  to,  while  other  times  costs  may  be  undefined. 
Costs  are  assigned  by  ccxnmodlty.  [Ref.  21] 

When  evaluating  additional  costs  Incurred 
It  Is  Important  to  look  at  where  efforts  are  expended.  If 
defects  occur  at  the  supplier's  facility  and  he  catches  th^, 
it  at  least  shows  that  his  system  works.  If  on  the  other  hand 
they  are  discovered  at  destination,  additional  costs  are 
Incurred  for  packing  the  item  euid  returning  It  to  the 
supplier.  It  also  raises  questions  ed>out  the  supplier's 
Internal  quality  control  system.  [Ref.  21] 

Costs  are  also  Incurred  by  withholding  a 
product  from  meuiufacturlng.  These  costs  are  hard  to 
determine;  an  arbitrary  cost  Is  assigned  normally  based  on 
overhead.  Additionally,  If  a  part  does  not  meet  the  quality 
requirements  for  Immediate  use  it  will  be  identified  with  a 
withholding  tag.  Costs  are  Incurred  for  additional  tests  that 
have  to  be  performed  before  the  cognlzcuit  officer  will  sign 
off  on  Its  use.  This  emphasizes  the  use  of  proper  product 
quality  certifications.  [Ref.  21] 

(3)  Lack  of  Performance  History.  Consider  ,t Ion 
must  be  given  to  new  suppliers  who  have  no  past  performcuice 
data  on  file.  Normally  at  Vought  Aircraft  they  request  a 
facilities  and  equipment  list.  They  ask  for  Information  on 
products  delivered  by  the  supplier  and  whether  the  product  Is 
on  a  Qualified  Products  List.  If  so,  to  what  process  was  It 
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approved?  If  a  supplier  claims  he  cam  perform  to  hot  forming 
specifications  but  has  no  approvals  It  could  be  cause  for 
suspicion.  The  compamy  may  even  go  Into  the  supplier's 
facility  to  see  what  the  process  looks  like  amd  ascertain  the 
quality  at  which  they  are  able  to  perform.  If  control 
processes  are  In  place  they  should  be  evaluated.  These 
processes  are  hard  to  get  amd  difficult  to  maintain.  To 
ultimately  be  placed  on  an  approved  supplier  list,  Vought 
looks  at  quality  assuramce,  technical  support,  mamagement 
support ,  procurement  support  amd  may  even  conduct  a 
capadsllltles  survey  depending  on  the  complexity  of  the 
product.  Once  Vought  Is  confident  In  the  supplier's  ad>illty 
they  will  be  sent  a  Request  for  Proposal  (RPP) .  [Ref.  21] 

When  a  manufacturer  is  unfaunlllar  with  a 
supplier's  past  performance  It  is  common  to  talk  with  Industry 
counterparts  to  see  how  well  a  supplier  is  doing.  If  they  are 
a  quality  supplier,  other  manufacturers  will  normally  tell  you 
so.  If  a  company  Is  non-ccxnmlttal  to  the  quality  of  a 
supplier,  it  Is  normally  an  Indicator  that  they  should  be 
avoided.  [Ref.  21] 
c.  Recency 

Mr.  Summit  noted  past  performance  Information 
cannot  be  used  if  it  is  more  than  three  years  old;  whether  it 
was  good  or  bad.  Facilities  and  equipment  wear  out  over  time. 
A  manufacturer  may  have  provided  a  quality  product  several 
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years  ago,  but  that  Is  no  guarantee  that  he  can  provide  the 
same  product  today.  [Ref.  21] 

There  are  also  considerations  other  than  time  to 
determine  if  a  coaQ)any's  past  performaince  should  be 
considered.  Such  situations  as  a  change  in  the  Chief 
Executive  Officer  (CEO) ,  a  high  turnover  of  quality  assurance 
personnel  or  a  sellout  could  affect  past  perforoance 
information  even  though  it  is  less  than  three  years  old.  [Ref. 
21] 

d.  Relevancy 

Vought  Aircraft  does  not  necessarily  look  for 
similar  products  to  determine  whether  past  perfornance  is 
relevant  to  a  current  purchase;  rather  they  look  at  the 
similarity  of  the  processes.  If  the  process  is  the  same,  the 
supplier  should  be  aUOle  to  produce  to  the  same  quality.  [Ref. 
21] 

e.  Problems  Associated  with  tbe  use  of  Past 
Performance 

Past  perfomance  evaluation  does  not  necessarily 
show  that  the  supplier  may  have  financial  problems  or  is 
pursuing  Chapter  11  procedures.  Additionally,  it  might  not 
reflect  that  the  supplier's  equipment  is  deteriorating  or 
wearing  out.  It  also  does  not  indicate  if  the  supplier  has 
the  capacity  to  meet  the  required  demand.  Suppliers  tend  not 
to  know  their  capacity  limits.  Equipment  may  not  meet 
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required  tolerances.  A  supplier  nay  have  the  dollars  to 
upgrade  equipment,  but  can  not  afford  to  stop  production  to 
fix  It.  [Ref.  21] 

Lastly,  Mr.  Summit  made  the  point  that  Vought  uses 
numerical  scoring  whenever  possible.  This  really  applies  to 
the  SPI.  When  audited,  agencies  lllce  to  see  as  little 
subjectivity  as  possible  cuid  am  e3q>lanation  of  the  thought 
process  In  selecting  a  supplier.  Vought  Aircraft  uses  good, 
fair,  euid  poor  to  rate  suppliers.  [Ref.  21] 

2.  McDonnell  Douglas;  Mesa,  AZ 

a.  Baclpground 

McDonnell  Douglas  sanufactures  the  AH- 64  Apache 
Attack  Helicopter,  the  MD  Explorer  amd  MD-500  helicopter  [Ref. 
l:p.  20] .  Mr.  Dave  Colemam,  the  Director  of  Procurement,  was 
Interviewed  to  develop  am  understamdlng  of  the  procedures  In 
place  at  McDonnell  Douglas. 

b.  Assessing  Past  Performance 

(1)  Supplier  Rating  System.  McDonnell  Douglas 
relies  on  their  Supplier  Rating  System  (SRS)  to  evaluate  past 
performance.  The  SRS  Includes  such  factors  as  quality  amd  on 
time  delivery  to  Include  the  (juality  of  paperwork  accompamled 
with  a  product.  Quality  Is  broken  out  Into  source  Inspection, 
receiving  Inspection,  and  In  line  rejections.  It  Is  a  running 
record  of  how  well  a  supplier  Is  doing  by  category  of  product. 
They  rate  suppliers  based  on  a  rolling  average  of  their 


39 


previous  12  months  experience  and  if  required,  can  go  back 
three  years  to  gather  data.  A  factor  Is  applied  automatically 
to  a  supplier's  proposal  based  on  data  from  the  rating  system 
to  account  for  poor  quality  or  late  deliveries  to  ensure  a 
best  value  selection.  For  suppliers  that  have  no  record  of 
prior  deficiencies,  no  additional  points  are  added.  Bach 
supplier  receives  a  report  card  four  times  a  year.  [Ref.  8] 

McDonnell  Douglas'  next  phase  to  the  SRS 
is  evaliiating  responsiveness:  how  well  a  compemy  responds  to 
their  needs.  With  a  smaller  supplier  base,  they  are  looking 
for  suppliers  that  have  100%  quality,  100%  delivery  and  are 
willing  to  work  with  them  on  a  partnering  basis.  The  drawback 
with  this  criterion  is  that  it  is  very  subjective  in  nature. 
[Ref.  8] 

(2)  Procurement  Review  Board.  For  larger 
procurements,  those  greater  than  $500,000,  a  Procurement 
Review  Board  is  convened  to  select  a  supplier.  The  board 
evaluates  the  proposal  for  best  value  to  include:  technical 
weighting,  price  weighting  and  mcmagement  weighting.  [Ref.  8] 

(3)  Certified  Suppliers.  During  the  source 
selection  evaluation  process,  recogn..tion  is  given  to 
certified  gold,  silver,  and  bronze  suppliers.  Additional 
points  are  awarded  to  those  suppliers  that  have  received  such 
ratings  based  on  their  past  performance.  To  receive  a  gold, 
silver,  or  bronze  certification,  a  supplier  has  had  to  go 


40 


through  an  assessment  for  his  type  of  bxislness .  The 
supplier's  processes  are  reviewed,  to  Include  his  Statistical 
Process  Control  (SPC) ,  auid  am  assessment  Is  made  of  how  well 
he  Is  doing.  [Ref.  81 

3.  Lockheed,  Ft.  Worth,  TZ 
a.  Background 

Products  at  Lockheed,  Ft.  Worth  Conpany  Include 
the  F-16  Fighting  Falcon;  F-111  modernization  emd  support  emd 
the  F-22  development  program  [Ref.  l:p.  10]  .  Mr.  Fred 
Ashed>ranner,  the  Chief;  Production  Equipment  Procurement,  was 
Interviewed  to  develop  an  understanding  of  the  procedures  In 
place  at  Lockheed. 

Jb.  Assessing  Past  Performance 

For  Lockheed,  past  performance  Is  a  slgnlflccint 
evaluation  factor.  First,  a  supplier's  price  amd  technical 
ed>illties  are  evaluated.  If  they  have  equal  ratings,  past 
performance  is  assessed  to  see  what  they  have  or  have  not 
done.  [Ref.  4] 

Lockheed  has  estcd>lished  a  formal  tracking  amd 
rating  system  that  records  the  percent  of  on  time  deliveries 
and  the  number  of  rejections  that  may  occur.  The  rating 
system  concentrates  on  cost  and  technical  capabilities  to 
Include  quality  and  overall  performance.  [Ref.  4] 

Supplier  ratings  are  ongoing.  Reports, 
referencing  rejection  rates  and  delivery  schedule  adherence. 
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are  sent  to  buyers  weekly  for  review.  If  a  negative  trend  Is 
developing  they  point  It  out  to  the  supplier.  If  the  supplier 
does  not  Inqprove,  no  further  orders  will  be  placed  or  they  may 
eliminate  the  supplier  from  their  base  altogether.  [Ref.  4] 

Lockheed's  system  looks  at  cost  growth.  They  look 
at  quality  cmd  where  rejections  occur.  If  rejections  occur  at 
the  source,  the  In^ct  Is  different  thain  having  the  part 
delivered  and  pulled  off  an  aircraft  to  be  sent  back  to  the 
supplier.  These  occurrences  are  Included  In  the  supplier's 
evaluation.  Data  In  supplier  evaluations  are  not  always 
negative.  Lockheed  atten^ts  to  provide  positive  feedback  as 
well.  Lockheed  uses  the  following  adjectives  for  supplier 
performance:  outstanding,  above  average,  satisfactory,  below 
average,  and  unacceptcdale.  [Ref.  4] 

Suppliers  have  a  positive  attitude  about  the 
evaluation  system.  They  believe  It  Is  fair.  It  points  out 
problems  so  corrective  actions  can  be  taken.  In  fact.  It  may 
point  out  problems  that  are  not  necessarily  those  of  the 
supplier,  but  rather  are  problems  Internal  to  Lockheed.  [Ref. 
4] 

For  large  purchases,  Lockheed  forms  a  procurement 
committee  made  up  of  top  management  and  quality,  engineering, 
and  procurement  personnel.  The  procurement  personnel  mctke  a 
presentation  of  data  and  facts  and  the  council  makes  the 
decision.  [Ref.  4] 
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For  raw  materials  and  small  purchases  Che  process 
Is  less  formal.  Normally,  the  buyer  emd  the  manager  make  the 
final  decision  as  to  the  source  selected.  Lockheed  requires 
suppliers  to  be  SPC  certified.  If  a  current  supplier  does  not 
become  SPC  certified,  they  will  eventually  be  dropped.  A  full 
time  SPC  group  evaluates  suppliers  regularly;  conducting  on 
site  audits  and  preparing  quality  reports.  [Ref.  4] 

Mr.  Ashabranner  en^hasized  that  in  this  new 
environment,  one  must  look  at  what  is  happening  at  the 
supplier's  facility.  What  changes  have  been  made  in 
personnel?  With  downsizing,  Lockheed  is  often  finding  that 
suppliers  are  keeping  their  best  people  and  that  there  are  no 
sharp  decreases  in  the  levels  of  efficiency  they  had 
previously  maintained.  In  other  cases,  the  most  experienced 
people  are  leaving  through  early  retirement  programs.  A 
supplier  may  be  an  old  conpany,  but  if  there  are  a  lot  of  new 
employees  there  will  be  an  effect  on  the  learning  curve.  [Ref. 
4] 

Many  suppliers  have  become  different  conpanies. 
They  have  diversified,  looking  for  a  larger  business  base. 
With  these  companies,  Lockheed  may  find  that  they  are  no 
longer  the  largest  buyer,  consequently  they  may  not  get  the 
Scune  priorities  they  had  previously  enjoyed.  As  a  result, 
time  delays  may  occur.  [Ref.  4] 

When  asked  about  whether  suppliers  tend  to  take  on 
more  work  than  for  which  they  have  the  capacity,  Mr. 
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Ashabrazmer  stated  that  many  suppliers,  especially  those  nrhere 
defense  Is  a  large  part  of  their  business,  currently  have 
excess  capacity. 

Mr.  Ashabrauiner  also  emphasized  that  the  goal  of 
quality  assurance  Is  for  the  supplier  to  have  a  high  enough 
rating  that  Lockheed  cam  reduce  the  degree  of  oversight  emd 
thus  reduce  costs. 

c.  Probleaa  Asaociated  with  the  Uae  of  Paat 

Performance 

With  the  chamges  In  the  defense  Industry,  It  may 
be  difficult  to  ascertain  a  ccmpainy's  past  performamce. 
Conpamles  are  going  out  of  business,  causing  the  prime  to  go 
elsewhere.  Suppliers  are  selling  out  amd  being  acquired  by 
larger  con5)anies  that  shut  the  facility  down  and  move  the 
operation  to  another  location.  This  requires  am  entirely  new 
survey  In  order  to  recertify  the  company's  SPC.  Sometimes, 
Lockheed  must  Ignore  previous  past  performamce  data  since  they 
are  evaluating  an  entirely  new  supplier  mamagement.  [Ref.  4] 

d.  Recency 

For  most  procurements,  recent  data  from  supplier 
evaluations  are  readily  available.  For  smaller  buys  however, 
performamce  data  may  be  two  to  three  years  old.  [Ref.  4] 

e.  Relevancy 

In  determining  whether  past  performance  is 
relevant,  it  frequently  depends  on  the  product.  For  products 
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previously  purchased  from  a  supplier,  they  look  at  how  well 
they  managed  the  last  requirement.  They  look  at  how  well  the 
supplier  came  on  line.  Additionally,  they  look  at  the 
supplier's  management  style,  the  philosophy  of  the  compemy, 
and  whether  the  supplier  has  a  good  track  record.  They  are 
looking  for  someone  that  will  be  committed  to  long  term 
relations.  Lockheed  is  concerned  with  the  processes  the 
supplier  employs  and  the  SPC  philosophy  pursued.  [Ref.  4] 

4.  Boeing  'ilrcraft;  Seattle,  WA 

a.  Backgrovnd. 

Products  at  Boeing  include  bomber,  tactical, 
tanker  and  airborne  early  warning  and  special  purpose 
aircraft;  offensive  cuad  defensive  mission  systems;  amd 
strategic  and  tactical  missile  systems  [Ref.  l:p.  6]  .  Mr.  Tom 
Roff,  the  Materials  Operations  Manager,  was  interviewed  to 
develop  an  understanding  of  the  procedures  in  place  at  Boeing. 

b.  Assessing  Past  Performance 

The  role  past  performcuice  plays  in  a  procurement 
is  dependent  on  the  dollar  value  and  conplexity  of  the  buy. 
Typically,  items  procured  fall  into  one  of  two  categories. 
The  first  is  low  dollar,  off-the-shelf  type  items.  Second, 
are  more  complex  units,  which  may  include  items  that  would 
require  machining  with  high  tolerances.  Additionally  included 
are  the  more  sophisticated  items  procured;  for  axaitple,  the 
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radar  for  the  Airborne  Warning  and  Control  System  (AWACS)  . 

[Ref.  19] 

(1)  Off-the-shelf  Items.  Historically,  for  vendor 
off-the-shelf  type  items,  price  has  been  the  primary 
determinate  for  source  selection.  Past  performance  is  now 
becoming  more  important.  Past  performance  data  are  evaluated 
from  the  last  two  years  with  a  12  month  rolling  average 
maintained.  A  supplier  added  index  is  assessed  to  the 
supplier's  proposal  to  account  for  the  costs  that  Boeing 
anticipates  it  might  incur.  This  forms  the  basis  of  award 
with  the  winner  being  the  supplier  that  provides  Boeing  with 
the  best  overall  value.  [Ref.  19] 

(2)  Procurement  of  Complex  and  Highly- 
Sophisticated  I^ems.  For  the  procurement  of 

these  items,  past  performance  is  very  iirportant.  Although 
price  is  a  consideration,  award  is  based  primarily  on  research 
and  development  capabilities,  technical  capabilities  and  the 
supplier's  management  capabilities.  Factors  evaluated  include 
the  supplier's  record  of  deliveries,  quality,  cost  control, 
risk,  technical  and  engineering  capabilities  and  his  ability 
to  integrate  his  product  with  the  rest  of  the  program.  Boeing 
also  evaluates  a  supplier's  responsiveness  to  corrective 
action  reports.  The  percent  of  weight  assigned  to  an 
evaluation  factor  is  commensurate  with  the  degree  of  risk  and 
complexity  of  the  procurement.  A  lot  of  emphasis  is  placed  on 


46 


capaUsllity,  including  the  personnel  in  the  supplier's 
organization.  Memy  facets  of  the  supplier's  evaluation  are 
objective,  while  others  are  siibjectlve,  especially  with  large 
procurements.  [Ref.  19] 

For  each  item,  Boeing  identifies  the 
in^ortcuit  characteristics  upon  which  the  supplier  should 
concentrate  his  efforts.  A  prerequisite  of  em  advauaced 
quality  system  is  the  reduction  in  the  variation  of  the 
characteristics  identified  by  Boeing.  The  supplier's  SPC  is 
key  to  controlling  variation  in  the  process  and  can  reduce  the 
number  of  inspections  required  for  delivery.  [Ref.  19] 

A  source  selection  board  is  often 
convened  at  Boeing.  The  number  of  members  on  the  board  and 
areas  represented  depend  on  the  conplexity  of  the  procurement. 
[Ref.  19] 

c.  Conpetition  Canalderations 

The  level  of  conpetition  also  determines  the  level 
at  which  past  performance  is  evaluated.  Sole  source 
procurements  offer  very  little  opportunity  to  evaluate  past 
performance  in  a  competitive  dimension.  On  the  other  hand,  a 
very  competitive  environment  offers  the  latitude  to  place 
greater  emphasis  on  how  well  suppliers  have  historically 
performed.  [Ref.  19] 
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d.  Problema  Aaaoci&^d  with  the  Use  of  Past 

Performance 

When  past  performamce  is  used  as  eui  evaluation 
criterion,  there  is  the  possibility  of  a  greater  number  of 
protests,  especially  if  award  is  not  placed  with  the  lowest 
offeror.  Additionally,  things  may  have  chcuiged  signiflcauitly 
since  the  last  contract  msUcing  the  use  of  past  perfomance 
less  relevant.  Mr.  Roff  also  pointed  out  that  if  you  place 
too  much  en^hasis  on  past  perfomance,  you  may  not  see  that  "a 
new  source  has  developed  a  better  mouse  trap" .  Lastly,  the 
FAR  has  historically  limited  the  eUoility  of  placing  greater 
enphasis  on  past  perfomance  because  of  the  thrust  to  aiirard 
contracts  based  on  price.  [Ref.  19] 

5.  Northrop}  Pico  Rivera*  CA 

a.  Background 

Products  at  Northrop  include  aircraft  and 
electronic  weapon  systems  [Ref  l.p.  20]  .  Mr.  Tom  Schoner,  the 
Small  Business  Liaison  Officer,  was  interviewed  to  develop  an 
understanding  of  the  procedures  in  place  at  Northrop. 
Tldditionally,  their  source  selection  procedures  guide  was 
provided  to  the  researcher. 

b,  ABBeaaing  Past  Performance 

As  with  other  commercial  organizations 
interviewed,  Northrop  estcQ^lishes  an  evaluation  panel,  a 
procurement  review  board,  cuid  a  source  selection  board  for 
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proposal  evaluation  auid  ultimate  source  selection.  In  the 
source  selection  procedures  used  by  Northrop,  e3q)erlence  and 
past  performance  are  sub-factors  of  management.  [Ref.  17 :p. 
23] 

(1)  Ei^erience.  In  the  area  of  e3q>erlence, 
Northrop  makes  a  determination  of  the  offeror's  normal 
Govenunent  euid  commercial  business  euid  how  It  relates  to  the 
proposed  work.  Also,  what  Is  the  offeror's  reputation  In  the 
field  to  which  the  proposed  work  relates.  [Ref.  17 :p.  23] 

(2)  Past  Performance.  Northrop  makes  a 
determination  of  whether  the  bidder  or  offeror  has  held 
previous  contracts  with  the  buyer  or  cmother  Government 
agency.  If  so,  the  results  are  evaluated,  particularly  the 
degree  to  which  the  supplier  has  adhered  to  schedules. 
Additionally,  If  technical  problems  arose  during  the  course  of 
the  contract,  were  they  solved  internally  or  was  there  heavy 
reliance  on  the  buyer's  technical  staff.  Northrop  also 
determines  If  there  was  an  unusually  high  number  of 
contractual  problems  that  may  have  resulted  from 
inflexibility,  naivete,  or  a  lack  of  cooperation  on  the  part 
of  the  supplier.  Lastly,  were  cost  over- runs  involved?  Were 
they  attributed  to  initial  low  estimates  or  as  a  result  of 
problems  encountered  during  the  course  of  the  contract  that 
could  not  have  been  anticipated?  [Ref.  17 ;p.  23] 
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c.  Recency 

The  age  of  past  performance  data  evaluated  for  a 
procurement  depends  on  the  sophistication  of  the  product.  If 
eui  Item  is  a  non-production,  non-flyaUble,  it  is  not  quite  as 
critical.  If  however  it  is  a  major  system,  Northrop  will 
review  a  supplier's  past  performcmce  quite  extensively,  going 
back  several  years  if  required.  [Ref.  20] 

d.  Relevancy 

When  an  item  is  procured  for  which  Northrop  has  no 
past  performance  data  they  will  ask  if  the  item  had  been 
produced  for  someone  else.  If  so,  they  will  evaluate  how  well 
the  supplier  performed  for  these  other  contractors.  They  also 
evaluate  performance  for  similar  type  items.  The  degree  of 
past  performance  data  sought  will  be  dependent  upon  the  level 
of  risk  measurement  associated  with  a  product.  [Ref.  20] 

C.  ANALYSIS 

The  coit^anies  surveyed  all  recognized  the  inqportance  of 
using  past  performance  in  the  source  selection  process.  The 
weighting  of  past  performance  is  commensurate  with  the  level 
of  complexity  of  the  item  being  procured.  The  more  conplex  an 
item,  the  greater  weight  past  performance  is  assigned. 

The  procedures  used  by  commercial  organizations  are 
similar  to  the  procedures  used  by  DoD.  The  underlying  theme 
promoted  by  both  is  best  value.  Past  performance  is  just  one 
facet  of  best  value. 
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The  evaluation  factors  used  to  assess  past  performamce 
were  common  among  the  conpanies  surveyed.  Quality  and  on- time 
delivery  were  predominate.  Quality  was  measured  by  a 
supplier's  ability  to  deliver  a  product  that  meets  the 
customer's  expectations.  It  must  be  evaluated  by  commodity, 
comparing  rates  to  those  of  the  supplier's  peers.  Boeing  made 
the  point  that  they  identify  the  characteristics  to  which  they 
desire  suppliers  to  pay  particular  attention.  In  determining 
a  supplier's  cQaility  to  meet  quality  requirements,  caii^>aLnies 
surveyed  noted  that  they  evaluate  the  processes  at  a 
supplier's  facility,  specifically  the  statistical  process 
controls  that  are  in  place.  On-time  delivery  was  measured  by 
a  supplier's  eODility  to  meet  schedule  requirements.  Companies 
surveyed  were  not  only  interested  in  the  product  delivered, 
but  were  additionally  concerned  that  all  accompcmying 
paperwork  and  certification  were  accurate. 

The  primary  discriminator  between  commercial  auid  DoD 
procurement  practices  appears  to  be  with  the  level  of 
evaluations.  The  coirpanies  surveyed  focused  attention  on 
their  supplier  rating  systems.  Not  only  do  the  data  from 
evaluations  become  the  basis  for  assessing  past  performance  in 
future  procurements,  but  they  serve  as  a  communication  tool 
between  the  supplier  and  the  conpany.  Evaluations  help  ensure 
a  quality  product  is  delivered,  while  at  the  same  time, 
provide  information  that  may  be  useful  to  the  conpany's  own 
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Internal  operations.  Suppliers  have  had  positive  feedback  on 
the  use  of  evaluations. 

The  procedures  In  place  In  the  coopanles  surveyed  mirror 
procedures  used  \inder  the  Navy's  RYG  Program.  Organizations 
are  concerned  with  the  costs  associated  of  doing  business  with 
a  supplier.  They  keep  a  record  of  the  additional  costs,  e.g. 
rework,  that  are  Incurred  under  a  particular  contract.  In 
particular,  they  look  at  where  addltloxial  efforts  are  expended 
which  may  be  eui  Indicator  of  supplier  or  conpemy  problems.  A 
supplier's  proposal  Is  adjusted  to  reflect  the  e3q>ected  cost 
of  doing  business.  Supplier's  adjusted  prices  then  become  the 
basis  for  contract  award  with  the  winner  being  the  supplier 
that  Is  eUble  to  provide  the  best  overall  value. 

All  companies  surveyed  indicated  that  past  performance 
data  used  to  assess  risk  should  be  no  more  than  three  years 
old.  Anything  older  would  not  be  a  true  indicator  of  how  well 
a  supplier  could  perform.  For  items  purchased  on  a  recurring 
basis,  however,  conpanles  require  information  that  Is  no  more 
than  one  year  old.  Oftentimes,  assessments  are  made  based  on 
a  rolling  average  from  data  for  the  last  several  months. 
Companies  also  noted  that  the  volatility  of  the  defense  market 
will  have  an  Inpact  on  the  age  of  past  performance  data  that 
are  assessed.  Because  of  rapid  chcmges,  a  company  today  may 
not  be  the  same  conpany  It  was  a  year  ago.  Under  these 
circumstances,  past  perfoxmance  again  may  not  be  a  valid 
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Indicator  of  the  level  at  which  a  supplier  may  be  able  to 
perform  today. 

In  regard  to  relevancy,  cooqpanies  do  not  necessarily  look 
for  similar  products,  rather  they  look  for  similar  processes. 
If  the  processes  are  similar,  a  supplier  should  be  eUble  to 
mamufacture  to  the  same  quality  as  a  previous  product  for 
which  past  performamce  data  are  available. 

Problems  experienced  with  the  use  of  past  performauice 
varied  from  suppliers  not  knowing  their  o%m  capacity  limits  to 
the  overall  difficulty  of  being  able  to  ascertain  a  supplier's 
past  performance  because  of  a  rapidly  changing  defense  sector. 
No  conclusions  Ceui  be  made  in  regard  to  the  problems  one 
should  expect  with  the  use  of  past  performance. 

D.  SDmilUlT 

This  chapter  introduced  the  procedures  used  by  commercial 
organizations  that  have  to  follow  similar  regulatory 
requirements  of  Government  agencies.  An  analysis  was 
presented  depicting  similarities  of  procedures  used  and  imique 
initiatives  developed  by  the  aircraft  industry.  The 
information  from  this  chapter  will  be  used  in  euialyzing  the 
PPG  in  the  following  chapter. 
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ZV.  PAST  PBRPOnCMICI  QUIDS  (PPG)  AKALTSZS 


A.  ZSTSODUCTZQN 

This  chapter  Includes  an  amalysls  of  the  PPG  which  Is 
attached  as  Appendix  B.  Data  from  Chapters  II  and  III,  as 
well  as  secondary  source  Information,  are  used  for  emalysls. 
The  section  headings  depicted  correspond  to  sections  In  the 
PPG.  Most  sections  of  the  PPG  are  auialyzed,  except  those  few 
which  have  no  bearing  on  this  research. 

B.  PAMPHLET  APPLICATION 

Policy  Letter  92-5  requires  the  evaluation  of  past 
performance  on  all  competitively  negotiated  procurements  over 
$100,000.  The  PPG  however  states  that  the  procedures  outlined 
In  the  guide  are  optional  for  acquisitions  imder  $10,000,000. 

In  eui  interview  conducted  with  Mr.  Tappel,  the  CH47  and 
UH60  Production  Management  Brauich  Chief,  Aviation  and  Troop 
Commeuid  (ATCOM)  ,  it  was  noted  that  the  PPG  is  difficult  to 
adopt  to  smaller  procurements  since  many  of  the  con^)etitors 
may  not  have  past  experience  in  the  proposed  area  to  make  a 
risk  assessment.  This  is  the  issue  of  relevancy. 
Additionally,  the  PRAG  would  only  add  another  layer  of 
bureaucracy  to  a  smaller  dollar  procurement  that  would  seem  to 
be  unnecessary. 
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Currently,  approximately  5% -10%  of  the  procur^nents  at 
ATCOM  exceed  the  $10,000,000  threshold  [Ref.  22].  Stephen 
Kelman,  OFPP  Adxninistrator,  has  stated  that  one  of  the  two 
major  reforms  he  will  promote  will  be  the  increased  use  of 
past  performance  [Ref.  11  :p.  34]  .  The  procedures  described  in 
the  PPG  are  designed  for  large  procurements  auid  do  not 
necessarily  apply  to  those  procurements  that  make  up  a 
majority  of  a  buying  commcuid's  contract  actions.  The 
researcher  believes  that  with  the  increased  emphasis  being 
placed  on  meucimizing  the  use  of  past  performcuice  in  the  source 
selection  process,  a  viable  past  performance  guide  should  not 
be  so  narrowly  defined. 

C.  PRAG  APPROACH  VERSUS  PRE-AMARD  SURVEYS 

The  PPG  states  that  the  pre- award  survey,  conducted  by  the 
Defense  Logistics  Agency  (DLA) ,  is  used  to  determine  whether 
a  contractor  has  the  capacity  to  perform  a  contract  based  on 
the  analysis  of  many  areas.  Pre-award  surveys  are  "go/no-go" 
as  to  the  offeror's  ability  to  conplete  the  work.  A  past 
performance  survey  on  the  other  hand,  describes  the  degree  of 
confidence  in  the  work  being  performed.  [Ref.  2:p.  3] 

The  PPG  fails  to  discuss  the  PRAG's  cUDility  to  conduct  on¬ 
site  visits  which  are  also  different  from  pre-award  surveys. 
An  on-site  visit  could  be  used  to  resolve  issues  that  may 
surface  during  a  risk  assessment.  [Ref  22]  The  issue  arises, 
however,  whether  an  on-site  visit  at  one  facility  would 
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require  a  visit  to  all  the  offeror's  facilities?  This  would 
appear  to  be  especially  true  if  aui  on-site  visit  afforded  the 
offeror  am  opportunity  to  prove  he  has  over  cone  past 
perfonaamce  problems  and  would  put  him  in  a  better  stamding 
had  the  visit  not  occurred.  The  researcher  believes  that  am 
on-site  would  be  appropriate  amd  would  not  require  a  visit  to 
all  offeror's  facilities  as  long  as  the  visit  is  to  clarify 
and  gather  past  performance  data  and  would  not  result  in  what 
could  be  interpreted  as  meaningful  discussions. 

An  additional  issue,  not  discussed  by  the  PPG,  is  the 
contracting  officer's  responsibility  towards  overseeing  the 
actions  of  the  PRA6.  During  the  source  selection  process,  if 
an  offeror  is  considered  to  be  in  the  con^etitive  ramge,  the 
contracting  officer  shall  conduct  written  or  oral  discussions. 
The  content  amd  extent  of  the  discussions  is  left  to  the 
contracting  officer's  judgment.  [Ref.  10: Subpart  15.610]  If 
am  on-site  survey  were  conducted  during  discussions  to  clarify 
or  gather  additional  past  performance  data,  the  researcher 
believes  the  contracting  officer  should  oversee  their  actions 
to  maintain  the  integrity  of  the  system.  If  data  collection 
is  going  on  prior  to  determination  of  the  conpetitive  ramge, 
the  contracting  officer  must  make  clear  to  the  offerors  that 
an  on-site  visit  does  not  in^jly  inclusion  in  the  competitive 
range . 
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D.  PBAO  COMPOSITION  MSD  STKDCTOSI 

Camnercial  orgamlzations  interviewed  often  use  boards  to 
evaluate  and  in2Uce  a  source  selection  decision.  The  nuiober  of 
personnel  on  the  board  aind  the  hierarchial  level  of  the 
members  are  commensurate  with  the  dollar  value  of  the 
procurement  and  the  con^lexity  of  the  item  being  purchased. 
At  ATCOM,  the  PRAG  is  normally  chaired  by  eui  industrial 
specialist  with  the  remainder  of  the  group  consisting  of  a 
quality  representative  emd  an  individual  that  can  provide 
engineering  support  [Ref.  22] . 

The  PPG  permits  each  contracting  activity  to  determine  the 
appropriate  membership  of  the  PRAG.  It  recommends  including 
members  with  procurement,  cost,  and  technical  esqperience  as 
well  as  PRAG  experience.  The  size  of  the  PRAG  should  be 
related  to  the  number  of  contractors  and  subcontractors 
expected  to  be  evaluated.  Additionally,  the  nature  and 
complexity  of  the  procurement  will  have  a  bearing  on  the  PRAG 
composition.  [Ref.  2:p.  4] 

The  PPG  is  veiry  thorough  in  providing  guidcuace  on  the 
proper  conposition  of  the  PRAG.  The  guide  was  used  in 
determining  the  conposition  of  the  PRAG  at  ATCOM.  The  group 
is  able  to  satisfactorily  perform  its  recpiirement  of 
developing  a  risk  assessment  for  each  offeror.  [Ref. 22]  The 
PPG  is  also  consistent  with  comments  from  commercial 
organizations.  Specifically,  the  PRAG  conposition  is 
determined  by  the  level  of  conplexity  of  a  procurement. 
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B.  PAST  PBBPOSMAlfCB  SDB-FACTOBS 

Careful  consideration  must  be  given  to  the  factors  used  to 
evaluate  past  performance.  Although  certain  evaluation 
factors  may  be  relevant  to  s<mie  procurements  cmd  not  others, 
a  thorough  understauiding  of  all  factors  to  be  considered  not 
only  enhances  the  past  performance  assessment,  but  will  also 
help  in  developing  the  source  selection  plam. 

The  PPG  describes  the  use  of  the  offeror's  record  of  on- 
time  delivery,  good  technical  quality,  emd  cost  control  to 
determine  the  likelihood  of  success  in  meeting  the 
requirements  of  the  solicitation.  Additionally,  Policy  Letter 
92-5  suggests,  as  is  also  stated  in  the  PPG,  use  of  the 
contractor's  record  of  conforming  to  specifications,  steuidards 
of  good  workmanship,  record  of  containing  and  forecasting 
costs,  adherence  to  schedules  including  the  administrative 
aspects  of  performajice,  history  of  reasonable  and  cooperative 
behavior,  commitment  to  customer  satisfaction  euid  the 
contractor's  concern  for  the  interest  of  the  customer. 
Lastly,  Steven  Kelman,  has  gone  even  further,  stating  that  a 
contractor's  ability  to  solve  problems  without  litigation 
should  also  be  a  factor  in  a  past  performance  rating  [Ref  9:p. 
60] .  This  factor  should  be  contained  in  the  PPG. 

The  primary  factors  used  by  commercial  organizations  were 
quality  and  on-time  delivery  to  include  the  paperwork  required 
to  be  delivered  with  a  product.  Although  the  verbiage  may 
differ,  these  factors  are  included  in  the  PPG.  In  a  recent 


study,  approximately  100  Americeui  euid  British  manufacturing 
firms  were  surveyed.  The  results,  surveying  supplier 
performance  measures,  indicated  that  delivery  reliaOsility, 
quality  conformance,  and  response  flexibility  all  rated  higher 
than  price.  [Ref.  5:p.  26] 

The  Navy's  RYG  Program  focuses  predominately  on  quality. 
RYG  does  not  specifically  address  factors  but  rather  sets 
criteria  for  color  classifications.  These  criteria,  included 
in  Appendix  A,  can  support  quality  measurement  by  identifying 
reject  rates,  failed  first  article  tests,  and  the  number  of 
Quality  Deficiency  Reports  occurring  over  a  time  period. 

The  PPG  recognizes  that  orgeuiizations  have  wide  latitude 
in  selecting  evaluation  factors  and  sub- factors.  It  does  not, 
however,  recognize  that  factors  and  sub- factors  selected  must 
be  measurable  and  essential.  More  is  not  necessarily  better. 
Additionally,  the  factors  and  siob- factors  chosen  must  be 
independent  of  each  other  to  prevent  double  counting.  Lastly, 
it  is  important  that  factors  are  measured  against  measurable 
objective  standards  rather  than  only  conqparing  proposals  to 
each  other.  [Ref.  7;p.  775] 

As  noted  by  the  previous  paragraph,  the  PPG  fails  to 
address  the  development  of  standards.  Standards  serve  as 
indicators  of  the  minimum  level  of  performance  or  compliance 
to  meet  the  requirements  of  the  solicitation.  They  serve  as 
a  guide  to  determine  how  well  the  offeror's  response  meets  the 
Government's  goals.  The  minimum  acceptable  standard  must  not 
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exceed  the  requirements  of  the  specification,  nor  should  it 
address  requirements  not  addressed  in  the  solicitation.  [Ref. 
7:p.  7751 

F.  PAST  PERFORMANCE  NEIGHTING 

The  PPG  states  that  past  performance  should  be  given 
enough  weight  that  it  acts  as  a  valid  discriminator  between 
proposals.  Commercial  organizations  recognized  that  the 
weight  given  to  past  performance  should  be  commensurate  with 
the  complexity  of  the  procurement.  Evaluations  generally 
involve  quality,  delivery,  service,  euid  price  weighted  in 
accordance  with  the  requirements  of  the  purchase  [Ref.  6;p. 
35]  .  The  in^ortance  of  the  criteria  will  vary  from  one 
procurement  to  the  next  [Ref.  12 ;p.  32].  The  Analytical 
Hierarchy  Process  (AHP)  can  be  a  useful  tool  in  determining 
the  weight  to  assign  to  past  performance.  Basically,  the  AHP 
compares  evaluation  criteria  and  assigns  weights  depending  on 
the  degree  of  importance  of  one  criterion  to  the  next.  [Ref. 
12:pp.  32-33] 

The  Government  can  evaluate  an  offeror's  experience  and 
past  performance.  The  PPG  explains,  however,  the  importance 
of  not  exaggerating  the  emphasis  placed  on  past  performance  by 
evaluating  it  separately  and  then  again  as  an  aspect  of 
experience.  It  is  however,  appropriate  to  separate  personnel 
experience  from  company  experience  and  evaluate  both.  To  the 
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researcher  however,  the  distinction,  as  discussed  in  the  PPG, 
is  not  very  clear. 

The  GSA  (General  Services  Administration)  Handbook 
describes  past  experience  as  the  extent  of  the  offeror's  past 
experience  in  carrying  out  similar  work.  Past  performance 
refers  to  the  quality  of  the  offeror's  past  perfojmiance  in 
carrying  out  similar  work,  considering  timeliness,  cost 
control,  and  technical  success. 

The  GSA  Handbook  further  addresses  qualifications  of  key 
personnel  which  refers  to  the  availeUoility,  con^etency, 
pertinent  education,  and  related  e3q)erience  of  personnel. 
Personnel  experience  is  important  to  the  successful  coii5>letion 
of  the  contract.  Generally,  the  Government  should  address 
minimum  qualification  standards  in  the  solicitation.  [Ref. 
7:pp.  773-774] 

The  researcher  agrees  with  the  PPG  in  recognizing  that  the 
amount  of  weight  given  to  past  performance  should  be  enough 
that  it  acts  as  a  valid  discriminator  between  offers.  The 
PPG,  however,  fails  to  recognize  that  there  is  an  additional 
connection  between  the  weight  given  to  past  performance  and 
the  coir^lexity  of  a  procurement,  a  point  emphasized  by 
commercial  organizations.  Lastly,  it  is  important  to 
recognize  the  exact  difference  between  past  performance  and 
experience;  both  company  and  personnel.  The  GSA  Handbook,  as 
referenced  above,  does  a  good  job  of  describing  the 
differences  between  these  terms.  A  thorough  understanding  of 


61 


the  differences  between  past  performance  and  e3q>erlence  is 
necessary  in  ensuring  the  in^ortcuice  of  past  performamce  is 
not  exaggerated.  Lastly,  it  is  importcuit  for  the  SSBB  and  the 
PRA6  to  communicate  the  aspects  of  experience  emd  past 
performcuice  that  each  will  evaluate. 

G.  SATING  CATEGORIES 

The  PPG  uses  high  performance  risk,  moderate  performance 
risk,  low  performance  risk,  cuad  unknown  performance  risk  to 
categorize  risk  assessment.  The  Air  Force  uses  the  same  terms 
with  similar  definitions.  The  Navy's  RYG  Program  utilizes 
color  codes,  but  the  idea  of  dividing  the  level  of  risk  is 
still  generally  the  same.  The  PPG  further  recognizes 
sxibdividing  categories  to  further  enhance  the  conqparative 
analysis  of  offerors.  The  researcher  believes  the  PPG 
appropriately  identifies  the  risk  categories  and  is  consistent 
with  the  definitions  used  by  the  other  Services. 

H.  EVALUATING  NO  PAST  PERFORMANCE 

The  PPG  states  that  in  the  event  no  past  performance  data 
are  availcQale  for  an  offeror,  the  lack  of  data  should  be 
treated  as  an  unJcnown  performance  risk  that  is  neutral  and  has 
no  positive  or  negative  evaluative  significance.  One  of  the 
practices  used  by  commercial  organizations  is  to  ask  for 
information  on  products  delivered  by  the  offeror  and  whether 
the  product  is  on  a  Qualified  Products  Lists.  If  so,  to  what 
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process  was  it  approved.  [Ref.  21]  Even  though  this  appears 
that  it  may  have  application  to  non- developmental  items  only, 
if  the  processes  expected  to  be  used  are  similar,  they  may  act 
as  a  valid  indicator  of  eui  offeror's  eOaility  to  deliver  a 
quality  product  that  had  previously  not  been  developed.  If 
similar  processes  are  expected  to  be  used  it  is  possible  to 
use  past  performance  data  from  the  manufacturing  of  other 
products  in  making  an  overall  risk  assessment. 

The  researcher  questions  that  even  though  the  lack  of 
performance  data  results  in  assignment  of  a  neutral 
performance  risk,  how  is  a  winner  actually  determined  if  past 
performcuice  is  a  weighted  factor?  The  PPG  fails  to  discuss 
this  issue. 

The  researcher  believes  that  the  likelihood  of  an  offeror 
having  no  relevant  past  performance,  either  commercial  or 
Governmental,  is  relatively  small.  If  however,  it  does  occur, 
a  determination  must  first  be  made  as  to  whether  the  offeror 
should  be  included  in  the  competitive  range  based  on  his 
proposal.  If  the  offeror  is  not  included  in  the  conpetitive 
range,  the  lack  of  past  performance  would  have  no  bearing  on 
the  source  selection.  If  the  offeror  is  included  in  the 
conpetitive  range,  his  proposal  should  be  compared  to  other 
proposals  from  a  cost  and  technical  standpoint.  If  in  the 
end,  the  offeror's  proposal,  who  lacks  past  performeuice  data, 
is  determined  to  be  the  most  advantageous  to  the  Government, 
the  offeror  should  be  the  apparent  winner.  The  lack  of  past 
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performeuice  data  places  greater  enqphasis  on  the  pre-award 
survey  to  determine  the  offeror's  aUaility  of  meeting  the 
contract  requirements.  Lastly,  the  solicitation  should 
clearly  state  that  even  though  past  perfornance  is  a  weighted 
factor,  the  lack  of  past  perfornance  will  not  prohibit  award 
to  the  most  responsible  offeror  whose  proposal  is  the  most 
advantageous  to  the  Government. 

On  rare  occasions  the  Government  must  have  a  contractor 
with  a  proven  performance  record.  In  this  situation,  an 
offeror  lacking  proven  past  performance  may  represent  a 
moderate  or  high  risk  ranking.  When  experience  is  evaluated 
by  the  PRAG,  not  the  SSEB,  the  solicitation  must  clearly  state 
that  the  PRAG  will  evaluate  e3q>erience  as  well  as  past 
performance  and  the  lack  of  e3q)erience  may  result  in  a  high  or 
moderate  risk  rating.  [Ref.  2:p.  6]  The  researcher  interprets 
this  to  mean  that  if  the  Government  has  no  proven  past 
performance  data  to  evaluate,  but  the  offeror  has  past 
experience  in  the  expected  area  of  contract  perfornance,  a 
risk  rating  will  be  assigned  based  on  that  experience  versus 
a  moderate  or  high  risk  rating  occurring  if  the  experience 
were  ignored.  The  researcher  believes  this  to  be  fair.  If  am 
offeror  has  experience  in  the  expected  area  of  contract 
performance,  it  should  nullify  the  moderate  or  high  risk 
rating  that  would  occur  if  that  experience  were  ignored. 


Z.  WBXT  TO  ZKCLXJDB  IN  TBB  SOLICITATION 

The  solicitation  must  state  that  the  Government  will 
conduct  a  performemce  risk  assessment  based  on  the  past 
performance  of  the  offeror  and  their  proposed  subcontractors. 
Additionally,  the  Government  may  use  data  provided  by  the 
offeror  and  obtained  from  other  sources.  Lastly,  the  burden 
of  providing  thorough  past  performance  data  rests  with  the 
offeror.  [Ref.  2:p.  6] 

The  PPG  discusses  the  areas  in  the  cQjove  paragraph  in 
sufficient  detail.  The  previous  section  discussed  the 
situation  where  the  Government  must  have  a  contractor  with  a 
proven  performance  record  and  that  the  solicitation  is  to 
clearly  state  that  the  PRAG  will  evaluate  experience  as  well 
as  past  performance  cuad  that  the  lack  of  eaqperience  may  result 
in  a  high  or  moderate  risk  ranking.  Additionally,  the 
researcher  had  previously  recommended  that  the  solicitation 
clearly  state  that  even  though  an  offeror  lacks  a  proven  past 
performance  record  and  past  performance  is  a  weighted 
criterion,  the  offeror  whose  proposal  is  the  most  advantageous 
to  the  Government  will  be  awarded  the  contract  even  though  the 
offeror  may  lack  a  proven  performance  record.  The  PPG  should 
centrally  locate  solicitation  requirements  in  one  section  to 
ensure  none  have  been  missed  by  the  contracting  officer. 
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J.  SOOSCIS  or  SATA 

The  primary  sources  of  data  available  for  use  by  the  PRAG 
include  DLA  and  Army  Material  Commauad  (AMC)  activities. 
References  are  also  cited  by  the  offeror  which  may  include 
other  comnercial  organizations.  The  PPG  also  recognizes  the 
use  of  contract  files  which  contain  evaluations  for  all 
contracts  that  are  ejected  to  exceed  $100,000.  [Ref.  2:p.  7] 
The  researcher  believes  these  evaluations  are  critical  to  a 
risk  assessment.  They  provide  factual  data,  in  vrriting,  that 
is  not  lost  as  personnel  leave  agencies. 

Mr.  Tappel  noted  that  the  buying  agency  has  a  great  deal 
of  knowledge  about  the  past  performance  of  its  contractors 
since  they  deal  with  them  periodically.  The  CIS  data  are  not 
ais  comprehensive  as  the  data  available  in  the  buying  activity. 
He  also  recognized  that  contractor  evaluations  are  done 
poorly.  The  researcher  believes  the  evaluations  are  done 
poorly  because  there  is  no  incentive  for  the  contracting 
officer  to  spend  a  great  d^u  of  time  on  them.  Since  many  of 
the  contractors  are  dealt  xth  on  a  repetitive  basis,  the 
contracting  officer  already  has  thorough  knowledge  of  their 
performance  record  and  sees  no  value  added  in  a  con5)rehensive 
evaluation.  Just  as  contracts  are  not  closed  out  in  a  timely 
manner,  if  administration  is  maintained  by  the  procuring 
activity,  evaluations  are  given  very  little  time  because  of 
the  urgency  of  the  next  procurement. 
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An  additional  area  contained  in  this  section  includes  the 
recency  of  data  evaluated.  Recency  focuses  on  how  long  ago 
the  past  performance  that  is  being  evaluated  took  place.  In 
general,  the  more  recent  a  past  contract,  the  better  indicator 
it  is  of  the  expected  performauice  of  a  contractor  on  a  future 
contract.  Contracts  that  occurred  some  time  in  the  past  may 
not  be  a  valid  indicator  of  the  expected  performance  of  a 
contractor  right  now. 

Prior  to  OFPP  Policy  Letter  92-5,  OFPP  issued  a  draft 
policy  on  December  6,  1991  soliciting  comments  due  by  Jcuiuary 
6,  1992.  Agencies  responding  tended  to  feel  that  the 
retention  period  for  past  performance  data  should  be  for  a 
period  greater  than  three  years,  such  as  six  years  since 
contract  compliance  actions  often  cam  taOce  a  long  time  to 
resolve.  Industry  on  the  other  hand  indicated  they  do  not 
want  ratings  held  for  any  longer  than  three  years.  This  is 
perhaps  the  result  of  the  differences  in  the  two  environments. 
In  the  Government,  changes  tend  to  take  a  longer  time  to 
occur,  con^ared  to  the  commercial  sector  where  changes  can 
occur  quite  rapidly.  It  makes  sense  that  a  firm  that  may  have 
had  a  poor  rating  four  years  ago  does  not  want  that  to 
represent  their  company  today.  Changes  in  personnel, 
management  and  the  manufacturing  process  could  make  it  an 
entirely  different  organization.  [Ref.  25 ;p.  9] 

The  researcher  believes  the  ideal  retention  period  is  not 
only  divided  between  Government  and  industry,  but  is  also 
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divided  among  industry.  In  certain  sectors  of  industry, 
changes  in  one  ccmniodity  may  occur  quite  rapidly  as  compared 
to  others.  For  exanple,  the  ccmputer  industry  is  changing 
c[uit  rapidly,  whereas  the  airfrcune  industry  is  rather 
stagneint.  It  follows  that  the  organization  to  manage  a 
particular  commodity  is  going  to  chemge  as  the  production 
process  for  that  commodity  changes.  So,  to  say  that  three 
years  may  be  an  appropriate  retention  period  for  one  industry, 
may  not  hold  true  for  emother.  In  s(»ne  cases  more  than  three 
years  may  be  appropriate,  while  for  others  one  year  may 
suffice.  The  PPG  properly  identifies  that  the  retention 
period  should  be  determined  on  a  case-by-case  basis. 

Since  recency  is  discussed  under  this  section,  it  would 
also  be  appropriate  to  discuss  relevancy.  Relevamcy  focuses 
on  whether  the  past  performance  data  are  a  solid  indicator  of 
what  can  be  expected  for  future  performance.  The  more  similar 
the  product,  the  more  relevemt  the  past  performcuice  could  be. 
On  the  other  hand,  the  more  dissimilar  a  product  is,  the  less 
likely  the  risk  associated  with  the  e3q)ected  future 
performance  could  be  accurately  determined.  Commercial 
organizations  equate  relevamcy  to  the  similarity  of  the 
processes  used  to  manufacture  a  product.  Although  the  PPG 
discusses  relevancy,  it  never  recognizes  the  iir^ortamce  of 
looking  at  processes  to  determine  relevancy. 
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K.  USX  or  COMMIRCZAL  BSmBlCIS 

The  PPG  states  that  the  best  practice  is  for  the  PRAG  to 
rely  on  Government  information,  however  it  is  permissible  to 
use  nongovernmental  references.  There  may  be  cases  where  the 
Government  has  little,  if  any,  performance  history  on  an 
offeror.  In  these  circxjmstances,  it  may  be  necessary  to 
solicit  other  ccmmercial  orgeuaizations  for  which  the  offeror 
has  done  work  as  identified  in  response  to  Section  L  of  the 
solicitation.  [Ref.  2:p.  8] 

The  researcher  believes  there  is  nothing  wrong  with  using 
commercial  references  as  long  as  the  data  received  are 
objective  and  can  be  verified.  A  push  to  purchase  more 
commercial  type  items  may  increase  the  reliance  on  industry  as 
a  source  of  past  performance  data. 

L.  COLLECTING  INFORMATION 

The  PPG  states  that  the  PRAG  gathers  information  using 
(questionnaires,  telephonic  in<juires,  or  both.  It  identifies 
that  questionnaires,  although  useful,  provide  incomplete 
information.  The  PPG  recommends  using  a  (questionnaire 
initially  and  following  up  sources  with  promising  information 
with  an  interview.  [Ref.  2;p.  8] 

The  PPG  discusses,  quite  thoroughly,  how  to  conduct  cmd 
document  phone  interviews  and  develop  (questionnaires. 
Commercial  organizations  use  similar  procedures,  but  often 
follow-up  with  on-site  visits  to  nake  an  assessment  of  a 
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supplier's  expected  performance.  The  Government  conducts  pre¬ 
award  surveys,  however  these  are  only  conducted  at  facilities 
of  contractors  that  are  likely  to  receive  a  contract  award. 
The  PPG  fails  to  discuss  the  benefit  of  on-site  visits  as 
discussed  previously.  It  might  be  useful  to  conduct  an  on¬ 
site  visit  if  the  Government  is  unfcuniliar  with  the  facilities 
of  a  contractor  euid  is  uncible  to  make  an  assessment  of 
performance  risk  without  doing  so.  Additionally,  not 
mentioned  in  the  PPG,  but  periodical  research  (trade  journals) 
may  also  prove  useful  to  assess  the  expected  performamce  of  am 
offeror.  This  would  be  especially  useful  if  the  Government  is 
not  familiar  with  the  products  mauiufactured  by  the  offeror. 
Overall,  the  PPG  does  a  good  job  of  discussing  how  to  collect 
data  but  fails  to  mention  on-site  visits  auid  periodical 
research  that  may  prove  to  be  useful. 

M.  ASSIGNING  PERFOBMANCE  RISK  RATINGS 

The  PPG  states  that  the  PRAG's  assessment  is  usually  based 
upon  subjective  judgment.  It  is  not  a  mecheuaical  process.  The 
assessment  should  include  the  rationale  for  the  conclusions 
reached.  [Ref.  2:p.  11] 

Overall,  the  PPG  discusses  how  to  assign  rankings  in  a 
satisfactory  manner.  It  does  not,  however,  discuss  the  use  of 
standards.  The  researcher  believes  the  use  of  standards  would 
eliminate  some  of  the  subjectivity  involved  in  the  PRAG 
process  and  would  better  define  rating  categories.  The  Navy 
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developed  criteria  for  determining  the  color  classification  of 
a  contractor  based  on  previous  performance.  Similar 
procedures  could  be  included  in  the  PPG  to  eliminate  some  of 
the  subjectivity  and  reduce  the  likelihood  of  protests 
occurring . 

For  exaaaple;  a  reject  rate  of  15%  or  more  for  two  or  more 
rejected  lots  in  the  last  two  years  results  in  a  contractor 
being  assigned  a  "red"  rating  iinder  the  Navy's  RYG  Program. 
Similar  criterion  could  be  used  as  a  measure  of  a  high 
performance  risk.  Although  there  would  be  some  subjectivity 
involved  in  determining  the  cutoff  rates  for  each  risk 
category,  the  sxibjectivity  involved,  as  a  whole,  would  be 
reduced.  The  researcher  recommends  developing  rates  for 
different  commodities.  As  identified  earlier  in  an  interview 
with  Mr.  Summit,  what  may  be  an  acceptable  reject  rate  for  one 
commodity  may  not  be  an  acceptable  rate  for  another. 

Lastly,  an  additional  issue  not  discussed  by  the  PRAG  is 
how  to  assign  a  rating  to  a  teaming  arrangement  [Ref.  22]  . 
Oftentimes,  teaming  arrangements  will  focus  on  making  up  one 
team' s  weaknesses  by  another  team' s  strengths .  Should  the 
PRAG  evaluate  teams  as  a  whole  or  only  partially  depending  on 
the  expected  performance  areas  of  each  team  member?  The 
researcher  believes  that  under  a  teaming  arrangement,  the 
expected  areas  of  performance  for  each  team  should  be 
evaluated  since  this  may  have  been  the  primary  reason  for  the 
arrangement  occurring.  Evaluating  all  areas  of  past 
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performance  for  each  team  places  unnecessary  emphasis  on  areas 
where  a  team  member  is  not  expected  to  perform.  The  PRAG's 
emphasis  should  be  on  the  areas  of  expected  performance  in 
meUcing  an  overall  risk  assessment. 

N.  THE  ASSBSSMBMT  REPORT 

The  PPG  states  that  the  PRAG  should  provide  enough  data 
for  the  source  selection  authority  to  make  an  informed 
judgment.  In  an  interview  conducted  with  Mr.  Tappel,  it  was 
noted  that  oftentimes  the  SSA  will  ask  for  the  entire  PRAG 
assessment,  especially  to  get  a  better  understanding  of 
specific  past  performance  problems  [Ref.  22] . 

The  researcher  concurs  with  the  PPG  in  this  area.  The 
PRAG  conclusions  should  be  supported  by  facts  addressing  the 
areas  for  the  SSA  to  make  an  informed  decision.  The  amount  of 
data  provided  by  the  PRAG  should  be  commensurate  to  the  number 
and  severity  of  problems  discovered  and  the  impact  of  that 
performance  on  the  offeror's  ability  or  meeting  contract 
requirements.  Whether  the  SSA  should  be  provided  the  entire 
assessment  will  predominately  depend  on  the  severity  of  past 
performance  problems  and  the  ability  of  the  PRAG  to  summarize 
them  in  sufficient  detail  for  the  SSA  to  have  a  full 
understanding  of  what  had  occurred. 
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O.  DZSCLOSURK  OF  PSAO  PINDZHaS  TO  OFFBRORS 

The  PPG  states  that  in  fairness  to  the  offeror,  past 
performance  problems  should  be  addressed  during  discussions  if 
the  offeror  had  not  previously  been  apprised  of  these  problems 
euid  provided  an  opportiinity  to  respond.  This  process  would 
validate  information  gathered  during  the  risk  assessment 
process.  [Ref.  2:p.  12] 

The  researcher  believes  the  PRA6  should  disclose  negative 
past  performance  information  to  the  offeror  prior  to 
discussions.  This  affords  the  offeror  an  opportunity  to  rebut 
data  that  may  prohibit  him  from  being  included  as  part  of  the 
competitive  range.  The  only  time  it  should  not  be  disclosed 
prior  to  discussions  is  if  the  performance  data  would  have  no 
bearing  on  the  offeror  being  included  in  the  competitive 
range . 

The  researcher  believes  that  when  disclosing  data  used  by 
the  PRAG,  only  the  factual  data  used  should  be  discussed. 
This  point  is  not  addressed  in  the  PPG.  It  is  not  necessary 
to  divulge  how  the  PRAG  came  to  their  conclusions.  Offerors 
should  only  have  the  opportunity  to  respond  to  the  accuracy  of 
the  factual  data.  Although  the  extent  of  discussing  proposal 
deficiencies,  prior  to  award,  is  limited,  the  researcher 
believes  past  performance  data  evaluated  by  the  PRAG  are  an 
exception  since  they  are  basically  outside  the  scope  of  the 
proposal . 
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Disclosing  PRAG  findings  supports  an  argument  for  an 
evaluation  system,  similar  to  CPARS,  that  allows  the 
contractor  the  ability  to  respond  to  contract  deficiencies. 
Such  a  system  would  prevent  an  offeror  from  not  having  prior 
knowledge  of  how  well  a  Government  agency  may  have  thought 
they  performed  and  the  subsequent  assessment  they  would  likely 
receive  from  the  PRAG  if  the  evaluation  was  used  in  the 
future.  With  the  Navy's  RYG  Program,  a  contractor  is  fully 
aware  of  his  color  classification. 

The  PPG  also  addresses  performance  problems  arising  with 
respect  to  subcontractors.  OFPP  Policy  Letter  92-5  requires 
the  subcontractor's  consent  prior  to  disclosing  information  to 
a  private  party.  The  prime  contractor  is  considered  a  private 
party.  The  PPG  suggests  requiring  the  prime  to  submit  its 
subcontractor's  consent  along  with  the  proposal.  This  would 
alleviate  the  barrier  to  discussing  subcontractor  information 
during  discussions.  [Ref.  2:p.  12] 

The  researcher  believes  that  if  consent  is  given,  once  the 
Government  becomes  aware  of  negative  past  performance  data  on 
a  subcontractor,  the  prime  should  be  notified.  This  offers 
the  prime  the  opportunity  to  lect  another  s\ibcontractor 
allowing  enough  time  for  the  PRAG  to  do  another  assessment 
without  delaying  the  procurement.  A  controversial  issue 
arises  when  the  subcontractor  has  not  given  consent  to  the 
prime  and  past  performance  problems  are  discovered  that  could 
possibly  prevent  the  prime  from  winning  the  competition.  The 
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researcher  believes  in  this  particular  case  the  problem  should 
be  brought  to  the  attention  of  the  prime.  The  problem  should 
not  be  initially  disclosed,  rather  it  should  be  brought  to  the 
attention  of  the  prime  that  a  disclosure  statement  from  the 
stib- contractor  was  not  included  with  the  proposal  and  to 
verify  whether  this  was  a  mistake  or  if  the  sub -contractor  is 
purposely  not  permitting  disclosure.  ' If  it  was  purposeful, 
the  researcher  believes  that  if  the  Government  has  superior 
knowledge  pertaining  to  a  sub- contractor,  it  should  be 
disclosed  and  a  decision  left  to  the  prime  whether  a  new  sub¬ 
contractor  should  be  selected.  It  may  have  been  more 
appropriate  for  Policy  Letter  92-5  to  have  read  third  party 
rather  than  private  party  to  prevent  this  problem  from 
occurring. 

P.  TREATMENT  OF  PAST  PERFORMANCE  INFORMATION 

As  stated  in  the  PPG,  the  information  gathered  by  the  PRAG 
should  be  treated  as  sensitive  source  selection  information. 
An  interview  was  conducted  with  ATCOM,  referencing  the 
disclosure  of  past  performance  information.  Mr.  Tappel  noted 
that  it  would  be  appropriate  for  a  buying  activity  to  share 
the  factual  data  of  an  offeror  to  other  activities,  but  not 
the  assessment  made  by  the  PRAG  [Ref.  22] . 

The  researcher  believes  the  PPG  does  a  very  good  job  in 
discussing  this  area.  The  importance  of  safeguarding  source 
selection  information  is  paramount  to  the  integrity  of  the 


system.  The  records  maintained  by  the  PRAG  are  their  own 
assessment  and  should  not  be  disclosed  to  other  buying 
activities.  The  information,  if  it  were  disclosed,  may  not  be 
useful  since  it  is  developed  around  the  requirements  of  the 
solicitation  which  will  most  always  be  different  from  one 
procurement  to  the  next.  The  sharing  of  factual  data  after 
source  selection  should  be  encouraged  since  it  may  be  useful 
to  the  PRAG  assessment  process  at  another  activity  that  may 
have  very  little  performance  history  on  an  offeror  that  it  has 
not  dealt  with  previously. 

Q.  GUIDE  APPENDIXES 

The  appendixes,  in  many  areas,  are  an  application  of  the 
data  previously  presented  in  the  PPG.  An  analysis  of  all 
appendixes  is  not  necessary  as  many  of  them  are 
straightforward  or  an  analysis  of  the  data  has  been  completed 
in  previous  sections  of  this  thesis. 

1.  Appendix  A;  Sas^le  Section  M  Solicitation  Provision 
Appendix  A  contains  a  sample  Section  M  Solicitation 
Provision.  Section  M  identifies  all  factors  and  sub- factors 
that  will  be  considered  in  awarding  the  contract.  The 

researcher  believes  the  sample  provided  is  thorough  and 
includes  all  requirements  specified  in  the  FAR  [Ref 
10:15.605],  however  section  M.4(2){b)  which  identifies  the 
areas  upon  which  the  Government  shall  focus  its  efforts,  is 
unclear.  It  is  difficult  to  determine  if  these  are  all 
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Intended  to  be  sub- factors  emd  if  so,  their  relative 
importance.  The  researcher's  interpretation  of  the  FAR  is 
that  svib- factors,  as  well  as  evaluation  factors,  are  to  be 
listed  in  order  of  relative  iir5>ortcUice .  Additionally,  if  a 
sub- factor  is  listed,  it  is  expected  to  be  evaluated.  The 
researcher  recommends  including  only  the  sub- factors  that  will 
be  evaluated,  listed  by  relative  importance. 

2.  Appendix  F  and  J^pendlx  6}  Sa]q>le  Questions  and  Ideas 

Appendixes  F  and  G  are  very  similar  so  they  will  be 
discussed  concurrently.  Appendix  F  lists  guidelines  for 
conducting  telephone  interviews  and  formulating 
questionnaires.  Appendix  G  is  a  sait^le  questionnaire.  It  may 
be  appropriate  to  combine  the  two  appendixes  to  reduce 
duplication. 

Overall,  both  appendixes,  as  written,  provide 
excellent  guidelines  for  developing  interview  questions  and 
questionnaires.  It  is  important  however,  to  recognize  that 
the  interviews  conducted  or  questionnaires  developed  should 
support  the  goals  of  the  procurement.  Additionally,  the  depth 
of  these  activities  should  reflect  the  complexity  of  the  item 
being  purchased  and  the  dollar  value  of  the  contract.  These 
two  points  are  not  made  in  the  Appendix. 

From  data  gathered  through  the  researcher's  interview 
of  commercial  organizations  and  secondary  sources,  the 
following  questions  were  developed  and  could  be  included  in 
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the  PPG  as  part  of  what  Is  now  Appendix  G.  These  questions 
are  included  because  they  give  a  strong  perspective  of  what  is 
in^ortant  to  commercial  organizations  euid  may  be  useful  to  the 
PRAG  process. 

•  Were  there  any  disputes  over  the  course  of  the  contract? 
If  so,  was  litigation  used  as  a  maumer  of  resolving 
conflicts? 

Stephen  Kelmaui  recently  stated  that  the  adsility  of  a 
contractor  to  solve  problems  without  litigation  should  be  a 
factor  considered  in  developing  a  risk  assessment.  The 
researcher  concurs  with  Mr.  Kelman's  assessment  and  that  the 
only  way  it  will  be  implemented  is  through  documentation 
similar  to  the  PPG.  Additionally,  solving  problems  through 
means  other  than  litigation  can  reduce  overhead  at  a  time  of 
a  decreasing  defense  budget. 

•  How  well  was  the  contractor  able  to  integrate  his  product 
with  the  rest  of  the  program? 

This  question  is  considered  in  Boeing's  assessment  of  a 
supplier.  The  researcher  believes  it  is  in^ortant  to 
Government  procurement  as  well.  The  ability  of  a  contractor 
to  integrate  his  product  with  the  rest  of  the  program  often 
results  in  the  program  staying  on  schedule. 

•  Was  the  documentation  required  to  be  delivered  with  a 
contractor's  product  accurate  and  complete? 


78 


The  ability  of  a  supplier  in  meeting  a  delivery  schedule 
is  important  to  commercial  organizations.  Equally  inqportauit 
is  the  quality  of  the  paperwork  to  be  delivered  with  the 
product.  This  is  especially  true  in  the  aircraft  industry. 
Although  the  PPG  recognizes  the  inqportcuace  of  schedule 
adherence  cuid  administrative  matters,  it  is  unclear  if  it 
specifically  addresses  the  quality  of  paperwork  to  be 
delivered  with  a  product. 

•  What  feedback  did  the  contractor  have  concerning  any 

performance  evaluations? 

The  researcher  believes  it  is  important  to  document 
performance  problems  either  during  the  contract  or  immediately 
upon  completion  and  afford  the  contractor  the  aJaility  to 
provide  feedback.  Commercial  organizations  have  recognized 
the  importance  of  evaluations  and  allowing  supplier  feedback. 
Supplier  feedback  may  prove  that  problems  are  not  that  of  the 
supplier,  rather  are  problems  on  the  part  of  the  prime.  The 
feedback  provided  by  contractors  during  evaluations, 
particularly  in  areas  where  they  say  they  will  make 
improvements,  identify  areas  for  the  PRAG  to  focus  its 
efforts.  Additionally,  if  a  consensus  is  not  reached  during 
the  evaluation  process  between  the  contractor  and  the 
contracting  officer,  going  back  to  the  evaluation  and  noting 
the  comments  made  by  the  contractor  can  give  the  PRAG  a  better 
understanding  of  the  actual  performance  achieved. 
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•  If  defective  parts  were  manufactured,  where  were  they 
identified? 

In  interviews  conducted  with  comnercial  org«mizations, 
they  identified  the  inportcmce  of  noting  where  deficiencies 
are  found.  If  they  are  found  before  leaving  the  supplier's 
facilities,  it  at  least  shows  the  system  is  working.  If  they 
are  not  foxind  until  the  part  is  received  at  the  loading  dock 
of  the  prime,  not  only  are  additional  costs  incurred,  but  it 
questions  the  supplier's  systems  that  are  in  place. 

R.  SUMMARY 

This  chapter  analyzed  the  PPG  identifying  major  strengths 
and  weaknesses.  Overall,  the  PPG  is  an  excellent  document. 
It  provides  guidance  in  determining  the  appropriate 
composition  and  structure  of  the  PRAG.  It  identifies 
inportant  sub- factors  to  evaluate  in  making  cui  assessment  of 
past  performance.  The  PPG  discusses  the  appropriate  weight  to 
be  placed  on  past  performance.  The  rating  categories  depicted 
in  the  PPG  are  consistent  with  the  categories  used  by  the 
other  Services.  It  discusses  what  should  be  included  in  the 
solicitation  ouid  the  inportemce  of  the  recency  of  data 
evaluated.  The  PPG  discusses  procedures  for  conducting  and 
documenting  telephone  interviews  in  great  detail.  Lastly,  it 
identifies  the  importance  of  treating  PRAG  information  just  as 
euiy  other  source  selection  material. 
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To  the  researcher,  there  are  several  weaUcnesses  of  the 
PPG.  First,  it  is  narrowly  defined.  Additionally,  it  fails 
to  discuss  the  use  of  on-site  visits  and  the  responsibility  of 
the  contracting  officer  in  overseeing  PRAG  actions.  It  does 
not  address  the  use  of  steuidards,  nor  does  it  sufficiently 
address  how  a  neutral  rating  is  treated  in  the  overall  source 
selection  process.  The  PPG  fails  to  discuss  how  to  evaluate 
teaming  arrangements.  Lastly,  the  PPG  fails  to  identify 
looking  at  the  processes  used  by  an  offeror  in  determining  the 
relevancy  of  past  performance  data. 

The  strengths  and  weaknesses  identified  in  this  Chapter 
will  be  used  in  Chapter  V.  Chapter  V  contains  the 
researcher's  recommendations  and  conclusions  and  identifies 
areas  for  further  research. 
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V.  CONCLnSZQHS  AND  RBCOMMBNDATZONS 


A.  ZNTRODUCTZON 

This  chapter  will  discuss  the  conclusions  and 
recommendations  resulting  from  the  thesis  research.  After 
presenting  the  conclusions  aind  recommendations,  the  research 
questions  will  be  cuiswered.  Lastly,  recommendations 

concerning  areas  for  further  research  will  be  made. 

B.  CQNCLUSIONS 

1.  The  use  of  past  performcuice  in  the  source  selection 
process  is  fundamental  in  ensuring  a  successful 
procurement . 

Steven  Kelman,  OFPP  Administrator,  has  stated  that 
past  performance  will  be  one  of  the  two  reform  issues  he  will 
promote  [Ref  11:  p.  34].  The  Army's  implementation  of  the 
Performance  Risk  Assessment  Group  is  just  one  application  of 
past  performance  that  is  being  seen  across  the  Department  of 
Defense.  All  commercial  organizations  interviewed  recognized 
the  use  of  past  performance  as  a  critical  element  in  the 
source  selection  process.  As  the  industrial  base  gets 
smaller,  maximizing  the  use  of  past  performance  can  help  weed 
out  suppliers  with  a  poor  performance  history.  The  Defense 
Personnel  Support  Center  recently  iitplemented  efforts  to  place 
increased  emphasis  on  past  performance.  Their  default 
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terminations,  by  the  close  of  FY93,  had  dropped  nearly  90%. 

[Ref  15 :p.  12] 

2.  The  en^hasis  placed  on  past  performance,  in  both 
Government  and  industry,  is  commensurate  with  the 
level  of  complexity  of  the  item  being  procured. 

In  interviews  conducted,  with  both  Government  axxd 

commercial  organizations,  it  was  recognized  that  the  role  past 

performance  plays  is  unique  to  every  procurement.  For  off- 

the-shelf  type  items,  past  performance  may  have  little  or  no 

bearing.  For  more  complex  procurements,  past  performauace 

could  be  critical.  The  degree  of  emphasis  placed  on  past 

performance  is  commensurate  with  the  complexity  of  the  item 

procured.  The  relative  importance  of  past  performance  should 

be  discussed  during  the  development  of  the  source  selection 

plan. 

3.  Quality  and  on-time  delivery  are  the  areas  of  past 
performance  most  important  to  industry. 

Every  commercial  organization  interviewed  indicated 

that  quality  and  on-time  delivery  were  the  most  important 

areas  of  past  performance.  Not  only  was  on-time  delivery 

important,  but  the  quality  of  the  paperwork  to  be  delivered 

with  a  product  was  equally  as  important.  A  study  of  100 

American  and  British  manufacturing  firms  indicated  that 

quality  conformance  and  delivery  relicQjility  were  more 

importeuit  than  price  in  evaluating  a  supplier's  performcuice. 

4.  The  overall  utility  of  the  PPG  is  excellent. 
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Although  the  PPG  does  have  scxne  shortfalls,  its 
overall  utility  is  excellent.  It  provides  guidance  in 
determining  the  appropriate  composition  cuid  structure  of  the 
PRAG.  It  identifies  important  sub- factors  in  making  an 
assessment  of  past  performance  that  are  consistent  with 
factors  used  by  commercial  orgauiizations .  The  PPG  discusses 
how  to  assign  the  appropriate  weight  to  be  placed  on  past 
performance.  It  is  also  consistent  with  the  rating  categories 
used  by  the  other  Services.  The  PPG  discusses  what  should  be 
included  in  the  solicitation  cuid  the  importance  of  relevancy 
of  data  evaluated.  Lastly,  it  discusses  procedures,  in  great 
detail,  for  conducting  and  documenting  telephone  interviews. 

5.  There  are  a  nxunber  of  specific  areas  in  the  PPG 
requiring  inprovement. 

There  are  several  wecUcnesses  to  the  PPG.  First,  it  is 
narrowly  defined.  It  fails  to  discuss  the  use  of  on-site 
visits  and  the  responsibility  of  the  contracting  officer  in 
overseeing  PRAG  actions.  The  PPG  does  not  address  the  use  of 
standards,  nor  does  it  sufficiently  address  how  a  neutral 
rating  is  treated  in  the  overall  source  selection  process.  It 
fails  to  discuss  how  to  evaluate  teaming  arrangements.  Lastly, 
the  PPG  fails  to  identify  excunination  of  the  processes  used  by 
an  offeror  in  determining  the  relevancy  of  past  perfomance 
data. 

6.  The  use  of  past  performance  under  sealed  bidding  is 
restricted  to  making  a  responsibility  determination. 
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Current  legislation  limits  the  use  of  past  performance 
in  sealed  bidding.  In  interviews,  emd  research  conducted,  no 
material  became  available  that  permitted  the  use  of  past 
performeince  in  sealed  bidding  other  than  in  making  a 
responsibility  determination. 

7.  A  number  of  potential  problems  exist  with  the  use  of 
past  performance  as  an  evaluation  criterion. 

A  number  of  problems  can  occur  with  the  use  of  past 

performance.  First,  an  assessment  of  past  performaince  does 

not  always  indicate  the  current  abilities  of  a  contractor. 

Additionally,  it  may  not  reflect  that  the  contractor's 

equipment  is  deteriorating  or  wearing  out.  It  may  not  reflect 

a  contractor's  current  fineuacial  status.  Also  important  is  to 

recognize  that  in  today's  environment,  even  though  you  may 

have  past  performance  data  on  a  company,  it  may  not  be 

relevant.  Lastly,  there  is  the  possibility  for  a  greater 

nxamber  of  protests,  especially  if  award  is  not  placed  with  the 

lowest  offeror. 

C.  RECOMMENDATIONS 

1.  Incorporate  procedures  similar  to  the  Navy's 
Red/ Yellow/Green  Program  into  the  PPG. 

The  PPG,  in  its  present  form,  discusses  the  use  of  the 

PRAG.  The  use  of  past  performance  however,  can  have  much 

broader  applications.  The  Navy's  use  of  TEAS  reflect  the 

actual,  expected  costs  of  doing  business  with  a  contractor. 

This  is  similar  to  many  of  the  systems  used  by  commercial 
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organizations  in  adjusting  a  contractor's  proposal  to  reflect 
best  value.  By  adopting  a  similar  system,  a  procurement  would 
better  reflect  the  actual  costs  e3q>ected  to  be  incurred. 

2.  Buying  organizations  should  develop  standards  against 
which  to  measure  successful  and  unsuccessful  past 
performance. 

The  PRAG  does  not  rate  cui  offeror  as  either  a  "go"  or 
a  "no-go".  Rather,  they  measure  the  degree  of  risk  based  on 
an  offeror's  previous  performance  history.  The  contracting 
officer,  though,  is  responsible  for  maJcing  a  responsibility 
determination  of  cm  offeror  under  con^etitive  negotiations  and 
a  bidder  under  sealed  bidding  procedures.  It  is  in5)erative 
that  standards  be  developed  by  the  procuring  agency,  for 
making  a  responsibility  determination  based  on  past 
perf  ozmamce . 

3.  The  Army  should  develop  a  supplier  evaluation  process, 
similar  to  that  used  by  commercial  organizations. 

Every  commercial  organization  interviewed  has 

developed  some  kind  of  supplier  rating  system  (SRS) .  Not  only 

does  a  SRS  serve  as  a  data  bank  for  performance  history,  but 

it  identifies  areas  for  a  supplier  to  concentrate  his  efforts. 

It  also  serves  as  a  mechanism  that  forces  communication 

between  the  buyer  and  the  supplier.  OFPP  Policy  Letter  92-5 

requires  the  use  of  evaluations  on  all  contracts  over 

$100,000. 

Mr.  Tappel  noted  that  contractor  evaluations  are  often 
poorly  done.  A  greater  eir5)hasis  must  be  placed  on  the 
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expected  benefits  of  conducting  thorough  evaluations.  The  use 
of  contractor  evaluations  could  help  increase  the  level  of 
communication  between  the  Government  and  the  contractor.  It 
is  important  for  the  information  to  be  timely.  The  evaluation 
period  used  by  commercial  organizations  in  rating  suppliers 
varies.  The  same  would  hold  true  for  military  organizations. 

4.  The  Army  still  needs  a  guide  that  is  not  so  narrowly 
defined. 

The  procedures  described  in  the  PPG  are  designed  for 
large  procurements  and  do  not  necessarily  apply  to  those 
procurements  that  make  up  a  majority  of  a  buying  command's 
contract  actions.  With  the  increased  emphasis  being  placed  on 
maximizing  the  use  of  past  performance  in  the  source  selection 
process,  a  guide  needs  to  exist  that  addresses  smaller  dollar 
procurements;  those  between  $100,000  and  $10,000,000. 

5.  The  PPG  should  eit^jhasize  the  disclosure  of  negative 
past  performance  findings  prior  to  discussions. 

The  PRAG  should  discuss  negative  past  performance 

findings  with  the  offeror  prior  to  discussions.  This  affords 

the  offeror  an  opportunity  to  rebut  data  that  may  prohibit  him 

from  being  included  in  the  competitive  range.  The  only  time 

it  should  not  be  disclosed  is  when  the  data  have  no  bearing  on 

the  offeror's  inclusion  in  the  competitive  range. 

6.  The  PPG  should  include  the  use  of  standards  in 
assigning  risk  categories. 

The  use  of  standards  would  eliminate  some  of  the 
subjectivity  involved  in  the  PRAG  process  and  would  better 
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define  rating  categories.  Although  there  would  be  some 
subjectivity  involved  in  determining  the  cutoff  rates  for  each 
risk  category,  the  subjectivity  involved,  as  a  whole,  would  be 
reduced. 

7.  The  PPG  should  provide  guidance  on  what  to  do  if 
negative  past  performance  data  are  discovered  on  a 
subcontractor  who  has  not  signed  a  consent  statement 
for  disclosure  of  findings  to  the  prime. 

The  PPG,  as  a  result  of  Policy  Letter  92-5,  states 

that  negative  past  performance  findings  on  a  subcontractor 

cannot  be  disclosed  to  a  private  party.  The  prime  contractor 

is  considered  to  be  a  private  party.  The  PPG  must  provide 

guidance  on  this  issue  to  prevent  the  unfair  exclusion  of  the 

prime  from  the  competitive  range. 

8.  The  PPG  should  provide  guidance  on  how  to  rate  teaming 
arrangements . 

Teaming  arrangements  often  focus  on  making  up  one 
team's  weaknesses  by  another  team's  strengths.  Evaluating  all 
areas  of  past  performance  for  both  team  members  nullifies  what 
may  be  the  primary  reason  for  the  arrangement  occurring. 

On  the  other  hand,  ignoring  poor  performance  may  not  be  fair 
to  the  other  offerors.  The  PPG  should  provide  guidance  on  how 
to  rate  tecuning  arrangements. 

9 .  The  PPG  should  address  how  to  conduct  scoring  when  no 
past  performance  data  exist  on  an  offeror  and  yet  it 
is  a  weighted  factor. 

In  the  event  no  past  performance  data  exist  on  an 
offeror,  a  neutral  weighting  is  to  be  assigned.  When  a 
neutral  weighting  is  assigned,  how  is  the  offeror's  overall  evalxjaticn 
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to  be  compared  to  the  other  offerors  for  which  past 
performance  data  exist?  In  fairness  to  all  offerors,  this 
issue  should  be  addressed  and  guidance  provided  on  how  a 
winner  is  to  be  determined  when  past  performance  is  used  as  an 
evaluation  criterion. 

10.  The  PPG  should  discuss  the  ability  of  the  PRA6  to 
conduct  on-site  visits. 

The  PPG  fails  to  discuss  the  benefit  of  on-site 
visits.  It  might  be  useful  to  conduct  an  on-site  visit  if  the 
Government  is  unfamiliar  with  the  facilities  of  a  contractor 
and  is  uncdsle  to  make  an  assessment  of  performance  risk 
without  doing  so.  If  on-site  visits  are  conducted,  the  PPG 
should  provide  some  guidcuice  on  how  they  should  be  conducted 
and  the  responsibility  of  the  con  rac ting  officer  in 
overseeing  the  PRAG's  activities. 

11.  Appendix  A,  of  the  PPG,  should  note  that  Section  M  of 
the  solicitation  should  disclose  the  relative  weight 
of  the  sub- factors  used  to  evaluate  past  performance. 

The  FAR  requires  that  the  factors  to  be  used  in 

evaluating  a  proposal  are  to  be  listed  in  the  solicitation  by 

the  order  of  relative  importance.  The  same  holds  true  for 

s\ib- factors.  The  PPG  lists  a  number  of  sxib- factors,  but  it  is 

unclear  whether  they  are  listed  in  order  of  relative 

importance  and  are  to  be  evaluated,  or  are  listed  only  to  give 

the  offeror  an  idea  of  the  proposed  areas  to  be  assessed. 


D.  ANSWBRS  TO  THB  RBSBAKCH  ODBSTIONS 

1.  How  Ceui  the  Am^  maximize  its  use  of  past  performance 
in  the  source  selection  process? 

The  PPG  is  a  start  to  maximizing  the  use  of  past 
performance.  It  is  the  first  past  perform£mce  specific 
docximent  addressing  the  issue.  It,  however,  may  have  little 
application  to  procurements  between  $100,000  cuid  $10,000,000. 
The  Army  still  needs  a  guide,  that  is  not  so  narrowly  defined, 
to  maximize  the  use  of  past  performcmce. 

2.  miat  performance  measures  should  be  evaluated  for 
source  selection? 

The  two  most  widely  used  performcmce  measures  for 
evaluating  past  performance  include  quality  and  the  ability  of 
a  supplier  to  meet  the  required  delivery  schedule  to  include 
the  appropriate  documentation  that  is  to  accompany  a  product. 
Additionally,  Kelmcm  has  recognized  that  the  degree  to  which 
a  contractor  solves  problems  without  litigation  should  be  a 
factor  in  rating  past  performance.  Cost  control,  research  emd 
development  capabilities,  technical  capabilities,  emd  a 
contractor's  momagement  capcdsilities  are  also  inportemt  in 
developing  a  risk  assessment  based  on  past  performance. 

3.  What  problems  have  been  encountered  with  the  previous 
use  of  past  performance  for  source  selection? 

In  interviews  conducted,  interviewees  rarely  noted 

problems,  but  recognized  the  potential  for  problems.  An 

assessment  of  past  performance  does  not  always  indicate  the 

current  abilities  of  a:  contractor.  It  might  not  reflect  that 
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the  contractor's  equipment  is  deteriorating  or  wearing  out. 
Additionally,  it  may  not  reflect  a  contractor's  current 
financial  status.  It  is  also  important  to  recognize  that  in 
today's  changing  environment,  even  though  you  may  have  past 
performance  data  on  a  compainy,  it  may  not  be  relevant.  As  a 
result  of  downsizing,  sell-outs,  euad  other  factors  affecting 
organizations,  the  company  the  Government  had  previously  done 
business  with  may  not  be  the  same  conqpany  other  than  in  name. 
Lastly,  there  is  the  possibility  for  a  greater  niunber  of 
protests,  especially  if  the  award  is  not  placed  with  the 
lowest  offeror. 

4.  What  information  is  availedale  for  use  by  the 
Performcuice  Risk  Assessment  Group? 

The  information  available  for  use  by  the  PRAG  includes 
data  provided  by  the  offeror  and  data  obtained  from  other 
sources.  Sources  include  references  cited  in  the  offeror's 
proposal,  DLA  and  AMC,  and  information  contained  in  the 
Contractor  Information  System.  Evaluations  are  required  at 
the  conpletion  of  all  contracts  that  exceed  $100,000  and  are 
available  for  performance  assessments.  The  methods  used  to 
collect  data  should  be  commensurate  with  the  size,  content, 
and  complexity  of  the  procurement. 

5.  What  special  conditions  exist  in  the  source  selection 
process  when  past  performance  is  used  as  an  evaluation 
criterion? 

The  solicitation  must  specifically  state  that  past 
performance  will  be  used  for  all  con^etitively  negotiated 


procurements  that  are  expected  to  exceed  $100,000  except  where 
the  contracting  officer  determines  It  not  appropriate. 
Additional  statements  must  be  included  in  the  solicitation  as 
stated  in  the  PPG.  With  the  increased  use  of  past  performance 
emd  the  use  of  procedures  with  which  offerors  may  not  be 
feuniliar,  it  may  be  necessary  to  esqplain  procedures  during  the 
pre- solicitation  or  pre- proposal  conference. 

6.  How  can  past  performance  be  used  under  sealed  bidding 
procedures? 

Current  legislation  limits  the  use  of  past  performance 
in  sealed  bidding.  Past  performance  is  used  in  mcdclng  a 
responsibility  determination.  The  Navy  has  circumvented  this 
somewhat  with  its  RYG  Program.  The  RYG  Program  allows  award 
to  be  based  on  the  best  value  to  the  Navy  by  including  the 
expected  costs  of  doing  business  in  cm  offeror's  proposal. 

7.  What  modifications  are  necessary  in  order  to  mcocimize 
the  use  of  the  PPG? 


•  The  PPG  should  en^hasize  the  disclosure  of  negative  past 
performance  findings  prior  to  discussions. 

•  The  PPG  should  include  the  use  of  standards  in  assigning 
risk  categories.  The  standards  should  be  incorporated  in 
the  solicitation  or  in  the  PPG. 

•  The  PPG  should  provide  guidance  on  what  to  do  if  negative 
past  performance  data  are  discovered  on  a  subcontractor 
who  has  not  signed  a  consent  statement  for  disclosure  of 
findings  to  the  prime. 

•  The  PPG  should  provide  guidance  on  how  to  rate  teaming 
arrangements . 

•  The  PPG  should  address  how  to  conduct  scoring  when  no  past 
performance  data  exist  on  cm  offeror  and  yet  it  is  a 
weighted  factor. 
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•  The  PPG  should  discuss  the  aUslllty  of  the  PRA6  to  conduct 
on- site  visits. 

•  Appendix  A  should  note  that  Section  M  of  the  solicitation 
should  disclose  the  relative  weight  of  sub- factors  used  to 
evaluate  past  performance. 

B.  ABBAS  FOB  FDBTBBB  BBSBABCB 

1.  Development  of  a  Contractor  Evaluation  System  (CES) 
With  the  increased  use  of  past  performance  in  the 

source  selection  process  it  is  necessary  to  develop  a  system 
that  captures  a  contractor's  performance  history.  A  CES  would 
provide  a  better  source  of  data  for  use  by  the  PRAG  in 
developing  a  risk  assessment  emd  could  be  used  by  the 
contracting  officer  in  meUcing  a  responsibility  determination. 

2.  An  Evaluation  of  Legislation  that  Inhibits  the  use  of 
Past  Performance 

With  the  current  shift  in  naximizing  the  use  of  past 
performance,  what  legislation  limits  its  use?  The  contracting 
officer  is  given  some  latitude  in  using  past  performance  in 
competitive  negotiations,  but  is  limited  vinder  sealed  bidding 
to  using  past  performance  in  naking  a  responsibility 
determination.  What  changes  can  be  made  to  the  FAR  that  would 
increase  flexibility  while  at  the  same  time  not  leave  DoD  open 
to  an  increased  number  of  disputes  and  protests? 

3.  Contractor  Certification  and  Awards  Program 

How  can  DoD  develop  a  contractor  certification  program 
that  would  be  widely  used  and  would  serve  as  an  incentive  to 
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contractors  as  sc»nething  to  achieve?  What  are  the  drawbacks 
of  the  current  programs  in  use:  the  Air  Force's  Blue  Ribbon 
Program,  emd  the  Army's  Contractor  Performance  Certification 
Program?  In  addition  to  certification  programs,  is  there  an 
awards  system  that  could  be  developed  to  recognize  world-class 
suppliers?  A  certification  and  award  program  would  support 
DoD's  attempt  to  become  a  world-class  customer.  If  neither  a 
certification  program  or  an  awards  system  is  appropriate,  how 
or  should  DoD  atirard  vcontractors  for  their  efforts  other  than 
through  award  and  incentive  fee  contracts? 
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APPENDIX  At  RBD/TBLLON/GSXBN  CRITERIA 

The  Navy's  RYG  Program  uses  the  following  criteria  for  color 

classifications  [Ref.  16:p.  13]: 

A.  RED 

B  -  Method  C,  D,  and/or  E  in  effect 

C  -  Latest  quality  Pre-A%#ard  Survey  in  last  2  years  no 
award 

D  -  Latest  Product -Oriented  Survey  in  last  2  years 
unacceptable 

E  -  Latest  Quality  Systems  Review  in  last  2  years 
unacceptcUale 

F  -  Latest  Special  Survey  in  last  2  years  unacceptcUale 

G  -  Reject  rate  15%  or  more  for  2  or  more  rejected  lots  in 
last  two  years 

H  -  Latest  2  First  Article  Tests  in  last  2  years 

unsatisfactory 

J  -  2  or  more  Category  I  QDRs  in  last  2  years 

K  -  6  or  more  Category  II  action  QDRs  in  last  2  years 

0  -  Yellow  reject  rate.  Latest  5  or  more  lots  rejected  in 
last  2  years 

B.  YELLON 

C  -  Latest  quality  Pre- Award  Survey  in  last  2  years  award 
with  findings 

D  -  Latest  Product -Oriented  Survey  in  last  2  years 

acceptcUDle  with  corrections 

E  -  Latest  Quality  Systems  Review  in  last  2  years 

acceptedjle  with  corrections 

F  -  Latest  Special  Survey  in  last  2  years  acceptable  with 
corrections 

G  -  Reject  rate  6-14%  for  2  or  more  rejected  lots  in  last 
2  years 

H  -  Latest  First  Article  Test  in  last  two  years 
unsatisfactory 

J  -  1  Category  I  QDR  in  last  2  years 

K  -  3-5  Category  II  action  QDRs  in  last  2  years 

0  -  Green  reject  rate.  Latest  5  or  more  lots  rejected  in 
last  2  years 

P  -  Red  reject  rate.  Latest  5  or  more  lots  accepted  in 
last  2  years 
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c.  Qsns 


C  -  Latest  (juality  Pre-Atiiard  Survey  in  last  2  years 

acceptcJale 

D  -  Latest  Product  Oriented  Survey  in  last  2  years 

acceptable 

E  -  Latest  Quality  Systems  Review  in  last  2  years 

acceptcd)le 

P  -  Latest  Special  Survey  in  last  2  years  acceptadsle 

6  -  Reject  rate  less  than  6%  for  5  or  more  lots  in  last  2 
years 

H  -  Latest  two  First  Article  Tests  in  last  2  years 

satisfactory 

K  -  0-2  Category  II  action  QDRs  in  last  2  years  and  G 
applies 

P  -  Yellow  reject  rate.  Latest  5  or  more  lots  accepted  in 
last  2  years 
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Foreword 


For  many  years,  the  U.S.  Army  Materiel  Command  (AMC)  acquisition 
community  has  recognized  that  the  quality  oi a  contractor’s  performance  on  previ¬ 
ous  contracts  should  be  an  evaluation  factor  in  most  of  our  competitively  negoti¬ 
ated  acquisitions.  To  fully  realize  the  advantages  of  evaluating  past  performance, 
it  is  essential  that  we  consider  not  only  performance  information  that  offerors  may 
include  in  their  proposals,  but  also  information  obtained  firom  other  sources.  Uni¬ 
form  AMC  procedures  for  obtaining  and  evaluating  past  performance  information 
have  been  developed  and  successfully  implemented. 

This  pamphlet  is  designed  to  help  you  participate  in  the  evaluation  of  past 
performance  during  the  source  selection  process.  It  updates  the  guidance  devel¬ 
oped  during  an  initial  pilot  program,  and  contains  the  most  up-to-date  procedures 
for  policy  implementation.  These  procedures  are  based  in  large  part  on  the 
“lessons  learned’’  and  suggestions  submitted  by  personnel  firom  AMC  contracting 
activities  who  have  been  durectly  responsible  for  the  successful  implementation  of 
the  program.  This  pamphlet  is  an  evolutionaxy  document  that  will  change  and 
improve  with  your  practical  suggestions  and  (he  latest  revisions  to  Department  of 
Defense  and  Army  acquisition  policy. 

Keep  in  mind  that  policy  and  procedures,  no  matter  how  well  devised,  are 
no  substitute  for  innovative  thinking  and  good  judgment.  This  pamphlet  provides 
a  basic  blueprint  for  conducting  past  performance  evaluations  witbin  the  tradi¬ 
tional  source  selection  process.  You  should  use  it,  not  rigidly,  but  as  basic  guid¬ 
ance  to  help  you  evaluate  past  performance  and  award  contracts  to  those  contrac¬ 
tors  who  will  deliver  quality  pxt>ducts  and  services,  on  time,  and  at  reasonable 
prices. 

I  extend  my  personal  thanks  to  the  members  of  the  Performance  Risk 
Assessment  Group  (PRAG)  Committee  who  were  responsible  for  developing  and 
implementing  the  PRAG  program.  I  also  extend  thanks  to  the  Mcgor  Subordinate 
Command  (MSC)  representatives  who  actively  participated  in  the  PRAG  Work¬ 
shop  and  made  invaluable  contributions  to  the  drafting  of  this  pamphlet. 
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I 


'‘Contractiaarpasipetfor^^  p^perly  documented  and 
considered^  is  a  powe^iodt^iit^pmai^ 
and  ttrneltness  ^Defense  contracts,**  ' 

—  Report  qf  JdttU  tX^DGD  Component  Contractor 
Ferfoirmdnc^  Retdeto  System  Task  Force 
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Program  Overview 


What  Is  Unique  About  This  Program? 


hi«  pamphlet  describes  an  innovative  way 
to  evaluate  contractor  past  performance 
diiring  the  source  sele^on  process.  AMC 
contracting  activities  helped  to  develop, 
test  and  implement  this  approach  which 
results  in  smarter  procurement  decisions 
and  better  contracts. 

The  distinctive  feature  of  this  program 
is  that  it  iises  information  that  is  outside 
of  the  offerors’  proposals  to  evaliiate  past 
performance.  No  longer  must  contracting 
activities  rely  solely  upon  the  very  contrac¬ 
tors  being  evaluated  for  past  performance 
information.  Now  contracting  officers  can 
use  independent  sources  of  information  to 
determine  how  well  those  contractors 
performed  in  the  past.  A  thorough  evalua¬ 
tion  of  past  performance  identifies  the 


When  Does  This  Pamphlet 


relative  performance  risks  associated  with 
competing  proposals  and  thereby  serves  to 
ensure  that  awards  are  made  to  good 
performers  rather  than  to  jiist  good  pro¬ 
posal  writers. 

A  second  unique  aspect  of  this  program 
is  that  it  provides  for  an  independent 
group  of  evaluators,  called  the  Perfor¬ 
mance  Risk  Assessment  Group  (PRAG),  to 
evaluate  past  performance  separately  from 
the  proposal  evaluators.  This  approach  is 
simple  and  flexible.  The  traditional  source 
selection  process  remains  unchanged 
except  that  now  the  Source  Selection 
Authority  has  additional  information  from 
the  PRAG  that  will  result  in  better  award 
decisions. 


Apply? 


he  procediires  outlined  in  this  pamphlet 
apply  to  competitively  negotiated,  best 
value  procurements,  in  which  the  selection 
of  a  source  is  based  on  factors  other  than 
price  alone,  including  best  value  competi¬ 
tions  based  only  on  price  and  past  perfor¬ 
mance.  In  fact,  contracting  activities  can 
use  this  process  in  a  variety  of  procure¬ 
ments  including  services,  supply,  and 
research  and  development.  The  proce¬ 
dures  would  not  be  appropriate  in  negoti¬ 
ated  procurements  when  the  award  is 


based  solely  on  low  price,  and  they  must 
not  be  used  to  circumvent  the  Certificate  of 
Competenpy  procedures  for  small  bxisi- 
nesses. 

While  the  Office  of  Federal  Procure¬ 
ment  Policy  (OFPP)  Policy  Letter  92-5 
dated  December  30, 1992,  requires  the 
evaluation  of  past  performance  on  all 
competitively  negotiated  acquisitioius  over 
$100,000,  the  procedures  outlined  in  this 
pamphlet  are  optional  for  acquisitions 
imder  $10,000,000. 
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How  Does  This  Approach  Differ  From 
Pre-Award  Surveys? 


t  is  important  to  distinguish  past  perform 
mance  evaluations  finmpre-awa^ 
surveys.  The  Defense  Logistics  Agency 
conducts  pre-award  surveys  to  determine 
whether  a  contractor  is  responsible. 
Responsibility  is  a  broad  concept  that 
addresses  whether  a  contractor  has  the 
capability  to  perform  a  particular  contract 
based  upon  an  analysis  of  many  areas 
including  financial  resources,  operational 
controls,  technical  skills,  quality  assur¬ 
ance  and  past  performance.  These  sur¬ 
veys  provide  a  “yes/no,”  “pass^ail,”  or 
“go/no-go”  answer  to  the  question,  can 
this  ofieror  do  the  work? 

Unlike  a  pre-award  survey,  a  past 


performance  evaluation  during  the  source 
selection  inncess  is  a  very  specific  endeavor 
that  seeks  to  identify  the  degree  of  risk 
associated  with  each  competing  offeror, 
thereby  permitting  a  comparative  assess¬ 
ment  offers.  Rather  thfiui  asking 
whether  an  offeror  can  do  the  work,  a  past 
performance  evaluation  asks,  wiU  it  do  that 
woric  successfully?  In  short,  it  describes 
the  degree  of  confidence  the  government 
has  in  the  offeror’s  likelihood  of  success.  If 
properly  conducted,  the  past  performance 
evaluation  and  the  pre-award  survey  will 
complement  each  other  and  provide  a  more 
complete  picture  of  an  offeror  then  either 
one  could  by  itself. 


- PRAG  Structure, - 

Composition  and  Evaluation 


What  Is  The  Function  Of  The  PRAG? 


The  PRAG  is  responsible  for  conducting  the 
past  performance  evaluation  to  determine 
the  degree  of  risk  involved  in  accepting  a 
contractor’s  promises  of  performance. 

This  determination  is  called  performance 


risk.  The  PRAG  prepares  a  report  that 
describes  these  risk  assessments  and 
identifies  strong  and  weak  points  in  each 
offeror’s  past  performance. 


What  Is  The  Proper  PRAG  Composition  And 
Structure? 


Each  contracting  activity  detaminea  the 
appropriate  membership  and  structure  of 
itsPRAGs.  The  quality  of  the  PRAG 
report  depends  upon  the  quality  of  the 
PRAG.  Ideally,  the  membership  should 
include  individuals  who  have  procurement, 
cost,  and  technical  expertise  as  well  as 
PRAG  experience.  The  individuals  se* 
lected  should  also  be  capable  of  making 
soimd  and  impartial  judgments. 

The  heart  of  the  PRAG  assessment  is 
the  information  gathering  process. 

Through  questionnaires  and  telephone 
interviews,  the  PRAG  can  obtain  a  detailed 
and  useful  picture  of  an  offeror’s  past 
performance.  Because  of  the  importance  of 
the  information  gathering  process,  it  is 
absolutely  critical  that  PRAG  members 
have  the  ability  to  conduct  meaningful 
telephone  interviews.  They  should  also  be 
able  to  assimilate  voluminoxis  data,  exer¬ 
cise  sound  judgment,  arrive  at  condusions 
that  make  common  sense,  and  communi¬ 
cate  those  conclusions  effectively  both 
orally  and  in  writing. 

The  size  of  the  PRAG  should  reflect  the 


number  of  contractors  and  subcontractors 
expected  to  respond  to  the  solicitation  as 
well  as  the  nature  and  complexity  of  the 
solicitation  requirements.  Experience 
indicates  that  a  four-person  team,  indud- 
ing  one  administrative  assistant,  is  a 
reasonable  size  for  a  solidtation  with  three 
to  six  offerors.  The  best  practice  is  to  have 
at  least  two  members,  one  with  procure¬ 
ment  expertise  and  one  with  technical 
expertise,  on  each  PRAG  to  allow  for 
di^ogue,  brainstorming,  and  in-depth  fact 
finding. 

The  PRAG  structure  should  enhance 
its  ability  to  independently  evaluate 
performance  risk.  The  PRAG  may  operate 
separately  from  the  Source  Selection 
Evaluation  Board  (SSEB)  and  report 
directly  to  the  Source  Selection  Advisory 
Coimcil  (SSAC),  or  it  may  operate  as  a 
separate  SSEB  subgroup  that  reports  to 
the  SSEB  chairperson.  A  PRAG  assess¬ 
ment  plan,  like  the  sample  at  appendix  C, 
should  be  developed  early  in  the  process 
and  made  a  part  of  the  source  selection 
plan. 


What  Subfactors  Should  Be  Used? 


The  contracting  activity  has  wide  latitude  in 
selecting  evaluation  factors  and  subfactors. 
The  past  performance  subfactors,  if  any, 
need  not  mirror  those  of  the  proposal 
evaluation.  In  most  cases  the  PRAG  at 
least  considers  the  offeror’s  record  for  on 
time  delivery,  good  technical  quality,  and 
cost  control  to  determine  its  likeliho^  of 
success  in  performing  the  solicitation’s 
requirements. 


OFPP  Policy  Letter  92-5  suggests  that 
past  performance  include  the  contractor’s 
record  of  conforming  to  specifications  and 
to  standards  of  good  workmanship;  the 
contractor’s  record  of  containing  and 
forecasting  costs  on  any  previously  per¬ 
formed  cost  reimbursable  contracts;  the 
contractor’s  adherence  to  contract  sched¬ 
ules,  including  the  administrative  aspects 
of  performance;  the  contractor’s  history  for 
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reasonable  and  cooperative  behavior  and 
commitment  to  customer  satisfaction;  and 
generally,  the  contractor’s  business'like 
concern  for  the  interests  of  the  customer. 

Relevancy  should  not  be  described  as  a 
subfactor.  Relevancy  is  a  threshold  question 


when  considering  past  performance,  not  a 
separate  element  of  pa^  perfisrmance. 
Irrelevant  past  performance  should  not 
form  the  buis  of  a  performance  risk  assesa 
ment. 


How  Much  Weight  To  Give  Past  Performance 


p 


ast  performance  should  be  given  sufficient 
evaluation  weight  to  ensure  that  it  is  mean¬ 
ingfully  considered  throughout  the  source 
selection  process  and  will  be  a  valid  discrimi¬ 
nator  among  the  offers  received. 

The  government  can  evaluate  both  the 
offeror’s  experience  and  past  performance. 
However,  it  is  improper  to  exaggerate  the 


importance  of  past  performance  by  first 
evaluating  it  separately  and  then  again  as 
an  aspect  of  experience.  Simply  put,  nei¬ 
ther  past  performance  nor  experience 
should  be  evaluated  twice.  It  is  proper, 
however,  to  distinguish  company  experi¬ 
ence  from  personnel  experience  and  evalu¬ 
ate  both. 


What  Are  The  Rating  Categories? 


T 


he  PRAG  may  use  the  following  definitions  of 
performance  risk  to  describe  the  results  of  its 
assessment: 

•  High  Performance  Risk  •  Based  on 
the  offeror’s  performance  record,  significant 
doubt  exists  that  the  offeror  will  successfully 
perform  the  required  effort. 

•  Moderate  Performance  Risk  -  Based 
on  the  offeror’s  perfonnance  record,  some 
doubt  exists  that  the  <^eror  will  successfully 
perform  the  required  effort 

•  Low  Performance  Risk  -  Based  (m 


the  offeror’s  performance  record,  little 
doubt  exists  that  the  offeror  will  success¬ 
fully  perform  the  required  effort 

•  Unknown  Perfonnance  Risk  -  No 
performance  record  identifiable  (This 
category  is  optional.  See  *How  to  Evaluate 
No  Past  Performance,”  page  6). 

Note:  Each  of  the  hi^,  moderate,  and 
low  risk  categories  may  be  further  subdi¬ 
vided  to  enhance  the  comparative  analysis 
of  offerors. 
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How  To  Evaluate  No  Past  Performance 


In  most  eases  the  PRAG  will  find  some 
related  past  perfonnanoe  infiurmation  for 
each  contractor  and  subcontractor,  espe* 
dally  if  the  PRAG  applies  a  broad  inter¬ 
pretation  of  relevancy.  Occasionally, 
however,  a  PRAG  cannot  find  any  rel¬ 
evant  information.  In  those  cases,  con¬ 
tracting  activities  should  treat  an 
offeror’s  lack  of  past  perfonnanoe  as  an 
unknown  performance  risk  that  is  neu¬ 
tral,  having  no  positive  or  negative  evalu¬ 
ative  significance.  This  approach  allows 
the  government  to  evaluate  past  perfor¬ 
mance  in  a  manner  that  is  fair  to  newcom¬ 
ers. 

An  alternative  approach  may  be  used 
on  rare  occasions  when  the  government 


must  have  a  contractor  with  aproven 
perfomumce  record.  In  this  situation,  an 
offeror  with  no  related  past  perfonnanoe 
may  represent  a  hii^  or  moderate  perfor¬ 
mance  risk  to  the  contracting  activity.  This 
alternative  approach  should  cmly  be  tiaed  if 
experience  is  evaluated  by  tbe  PRAG,  not 
the  SSEB.  In  this  case,  the  solidtation 
should  dearly  state  that  the  PRAG  will 
evaluate  e:q>erience  as  well  as  past  perfor¬ 
mance  and  that  a  lack  of  experience  may 
result  in  a  hi|^  or  moderate  risk  rating. 
Even  here  the  government  can  ease  the 
impact  on  newcomers  by  including  lan¬ 
guage  in  the  solidtation  that  encourages 
them  to  team  with  proven  performers. 


What  To  Include  In  The  Solicitation 


The  solidtation  should  dearly  state  that: 

(1)  the  government  will  conduct  a  perfor¬ 
mance  risk  assessment  based  upon  the 
past  performance  of  the  offerors  and  their 
proposed  subcontractors  as  it  relates  to 
the  probability  of  successful  accomplish¬ 
ment  of  the  work  required  by  the  solidta¬ 
tion; 

(2)  in  conducting  the  performance  risk 
assessment,  the  government  may  use  data 
provided  by  the  offeror  and  data  obtained 
fircnn  other  sources; 

(3)  while  the  government  may  elect  to 
consider  data  obtained  firom  other  sources, 
the  burden  of  providing  thorou^  and 
complete  past  perfimnance  infimnation 
rests  with  the  offeror.  Appendix  A  con¬ 
tains  a  sample  Section  M  provision  for  use 
in  solidtations.  . , 


Section  L  of  the  solicitation  should 
instruct  offerors  to  submit  information 
gnngaming  contracts  and  subcontracts 
which  are  in  any  way  similar  to  the  work 
required  by  the  solidtation,  or  which 
offerors  consider  relevant  in  demonstrat¬ 
ing  their  ability  to  perform  the  proposed 
effort.  Also,  it  is  important  that  the  (^eror 
specifically  describe  the  work  that  its 
subcontractors  will  perform  so  that  the 
PRAG  can  conduct  a  meaningful  perfor¬ 
mance  lisk  assessment  on  each  significant 
subcontractor.  Appendix  B  contains  a 
sample  Section  L  provisicm  tar  use  in 
sdidtationa. 

Presolidtation  or  preproposal  oonfor- 
ences  should  explain  the  performance  risk 
methodology  to  ensure  that  offerors  undei^ 
stand  the  procees  and  its  overall  signifi- 


PRAG  Operatioiis 


How  To  Begin 


The  PRAG  chairpenon  should  hold  a  meeting 
of  PRAG  members  as  soon  as  possible  prior  to 
the  receipt  of  proposals  to  outline  the  PRAG 
process,  obtain  signed  nondisclosure  sCate- 
mmits,  and  distribute  the  evaluatkm  plan 
and  R^uest  for  Proposal  (RFP).  Iherequir* 
ing  activity  should  brief  the  PRAG  on  the 
technical  requirements  of  the  acquintkm. 

The  PRAG  chair  may  assign  each  PRAG 
member  an  (^fenir<s)  for  whom  they  will 


screen  the  available  data  to  select  the  most 
recent  and  relevant  references  for  in*depth 
feet  finding.  However,  some  contracting 
activities  prefer  to  assign  the  work  by 
fimctional  area  rather  than  IqroSeror.  In 
either  event,  the  PRAG  members  will  meet 
after  gathering  past  performance  informa- 
tion,  to  determine  the  perfiumance  risk 
ratings. 


What  Sources  Of  Data  Are  Available? 


AMC  established  a  centralized  networking 
^stem  to  provide  sources  of  past  perfar- 
iiianoe  mformation  to  the  PRAG.  This  device 
is  called  the  Contractor  Infonnation  System 
(CIS),  and  it  contaiiM  only  factual  inftnna- 
tion  pertaining  to  contractors.  ThedSis 
basically  an  electronic  telephone  book.  It 
does  not  contain  report  cards  on  contractors. 
Questions  pertaining  to  the  CIS  or  the  PRAG 
methodology  that  caimot  be  answered  by  the 
source  selection  hierarchy  at  the  MSC  may 
be  referred  to  the  AMC  Deputy  Chief  of  Staff 
for  Acquisition. 

Although  the  PRAG  may  consider  data 
available  from  many  sources  such  asDLA 
and  AMC,  its  main  sources  of  information  are 
often  the  references  cited  by  offerors  in  their 
proposals.  Upon  receipt  of  proposals  and  the 
AMC  report,  the  PRAG  will  determine  which 
of  the  offeror’s  past  contract  efforts  relate  to 
the  solicitation  requirements.  Althoui^ 
these  determinations  of  relevancy  are  judg- 
ment  calls,  it  is  helpful  fo  considw  the 
offeror’s  explanation  of  relevancy  contained 
in  its  proposal 


In  some  cases,  previous  contracts  as  a 
whole  may  be  similar  to  the  current  con* 
tract  edrile  in  others  only  portions  of  previ* 
ous  contracts  may  be  relevant. 

For  example,  the  government  uses  Ada 
software  language  in  many  different  sys¬ 
tems.  ff  a  solicitation  calls  for  the  develop¬ 
ment  of  Ada  s(ftware  for  an  aircraft  sys¬ 
tem,  the  contractor  might  identify  a  previ¬ 
ous  effort  where  it  developed  Ada  software 
for  a  satellite  terminal  The  government 
may  consider  that  previous  effort  to  be 
relevant  for  purposes  of  assessing  the 
contractor’s  abilify  to  develop  Ada  software 
even  though  the  underlying  system  is 
different  from  the  current  requirement. 
Another  example  is  the  evaluation  of  the 
ocmtractor’s  management,  planning,  and 
scheduling  of  subcontractora  on  a  past 
service  contract  for  a  current  production 
requirement  calling  for  integration  skills. 

The  PRAG  should  conskfor  the  most 
recent  data  available.  The  best  practice  is 
to  select  efforts  that  are  either  still  in 
progress  or  just  completed,  and  that  have 
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at  leaat  1  year  of  perfiMnnanoe  hiatory. 

The  actual  tune  ia  left  Manic  in  the 
aample  Section  L  praviaton  in  «nianHi»  B 
becauae  it  ahould  be  determined  by  the 
contracting  ofiBcer  on  a  caae*by<SM  baaia. 
However,  moat  activitiea  have  uaed  3 
years  witli  much  aucoeaa. 

It  ia  noted  that  OFPP  Poli<7  Letter  92- 
5  dated  December  30, 1992,  not  only 
requirea  the  evaluation  of  paat  perfor¬ 
mance  during  the  source  edection  prooett. 


but  also  requires  contracting  activities  to 
prepare  an  evaluation  of  orntractor  perform 
mance  at  the  time  the  work  under  the 
contract  IS  eofitpfotcd  for  each  contract  in 
eaceaa  of  $100,000.  These  latter  evalua¬ 
tions  are  then  placed  in  the  contract  file. 
The  FRAGs  ahould  use  them  during  the 
source  selection  process  to  help  arrive  at 
their  own  assessment  of  an  offeror’s  past 
perfonnance. 


Can  The  PRAG  Use  Commercial  References? 


he  best  practice  is  to  rely  on  government 
sources  of  information.  However,  it  is 
permissible  to  use  nongovernment  refer¬ 
ences  when  necessary.  The  PRAG  should 
verify  information  received  from  commer¬ 


cial  and  foreign  government  sources  to 
ensure  accuracy.  The  use  of  such  refer¬ 
ences  for  one  (^eror  does  not  require  the 
same  for  all  offerors  so  long  as  sufficient 
information  is  available  for  them. 


How  To  Collect  Information 


he  PRAG  gathers  information  using 
questionnaires,  telephonic  inquiries,  or 
both.  Field  experience  indicates  that 
questionnaires  provide  useful  but  incom¬ 
plete  information.  A  helpful  approach  is 
to  start  by  sending  a  commcm  question¬ 
naire  to  each  reference  and  to  conclude  by 
calling  those  who  respond  with  the  most 


promising  information.  Experience  indi¬ 
cates  that  whether  you  send  question¬ 
naires  or  not,  you  most  likely  conclude 
by  w»lli"g  the  reference  to  obtain  more 
detail  or  clarification. 

Samples  of  questions  for  telephone 
interviews  and  written  questions  are 
included  in  appendixes  F  and  G. 
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Where  To  Conduct  Telephone  Interviews 


FoUowmg  the  screening  of  previoua  contracts 
for  further  in-depth  review,  each  raAG 
member  should  send  questionnaires  and/or 
initiate  telephone  calls  to  the  identified 
references  for  those  efforts.  The  interview¬ 
ing  and  reporting  of  results  are  usually 
individual  efforts  conducted  hy  each  FRAG 
member.  However,  it  is  sometimes  helpful 
for  the  FRAG  to  collect  infivmatum  as  a  ' 
group  through  the  use  of  conference  calls. 

In  any  event,  the  environment  in  which  this 
work  is  done  significantly  impacts  both  the 
time  required  to  complete  this  portion  of  the 
process  and  the  quality  of  the  results.  These 
activities  are  hampered  severely  if  each 
FRAG  member  attempts  to  conduct  tele- 


jdrone  interviews  at  their  normal  work  site 
with  all  of  its  attmdant  interruptions, 
distractions,  and  security  risks. 

If,  on  the  othtf  hand,  the  FRAG  mem¬ 
bers  are  aUe  to  aaaemUe  as  a  group  for 
telephone  interviews,  they  will  be  aUe  to 
provide  considerable  reinforcement  and 
instant  feedback  for  one  another.  Each 
FRAG  member  should  be  able  to  devote 
their  undivided  attention  to  this  initial 
assessment  process.  Althou^  this  ap¬ 
proach  requires  a  secure  area  that  is  large 
enou^  to  accommodate  all  of  the  FRAG 
members,  the  resulting  benefits  are  signifi¬ 
cant. 


How  To  Conduct  Telephone  Interviews 


The  telephone  interview  process  is  an  art 

form.  Until  a  smooth  conversation  pattern  is 
developed,  it  is  an  inherently  uncomfortable 
situation  for  many  people.  Tlierewillbe 
some  difficulty  learning  how  to  start  a  tele¬ 
phone  interview,  keep  it  moving,  and  cover 
all  important  areas.  As  the  interviewing 
process  continues,  the  FRAG  member  usu¬ 
ally  uncovers  special  items  of  interest  that 
he  or  she  will  want  to  pursue  throu|^  follow 
up  calls. 

At  least  two  references  should  be  con¬ 
tacted  on  each  previous  contract  effort  se¬ 
lected  for  in-depth  review.  Additional  refer¬ 
ences  are  (dlen  identified  during  the  inters 
views.  Maximum  effectiveness  occurs  when 
the  expertise  the  FRAG  interviewer 


matches  that  of  the  reference. 

Frior  to  initiating  a  telephone  inter¬ 
view,  a  FRAG  member  should  gather  all 
available  information  on  a  specific  effort 
and  draft  a  list  (^questions.  There  may  be 
a  common  group  of  questions  for  all  offerors 
and/or  tailored  questions  for  each  offeror, 
depending  upm  the  circumstances.  These 
questions  can  either  be  sent  as  question¬ 
naires  to  each  reference  or  be  used  by  the 
FRAG  member  during  the  telephrae  inter¬ 
view. 

At  the  start  ot  each  telephone  interview, 
the  FRAG  member  should  exidain  the 
purpose  of  the  call  and  request  voluntary 
assistance  fimn  the  reference.  The  FRAG 
member  should  exfdain  that  he  or  she  will 


document  the  reeulta  of  the  oanvereatum 
and  send  a  copy  of  the  memorandum  to 
the  refiwence  for  vwification.  There  ia 
usually  no  need  to  divulge  the  aoliatation 
numbw,  program  deacriptiaa,  or  other 
identiQdng  informatioa  to  the  reference. 

If  srott  do  80,  you  need  to  obtain  a  nondis¬ 
closure  statement. 

In  most  instances  the  refisrence  will 
willinfl^  provide  the  information  re¬ 
quested.  bi  those  rare  cases  when  the 
reference  refuses  to  particqtate,  the  FRAG 
member  should  request  assistance 
throuf^  the  source  selection  hierarchy  at 
the  MSG.  Alternatively,  the  PRA.G  mem¬ 
ber  may  attempt  to  continue  the  interview 


off-the-record  to  obtain  data  that  may  be 
validated  on-the-record  during  interviews 
with  other  reforences. 

It  is  important  to  pursue  the  underiy- 
ing  facts  supporting  any  conclusion  ary 
statements  received  on  a  contractor, 
particularly  if  they  are  unusually  positive 
or  negative.  The  FRAG  member  can 
determine  neither  the  magnitude  of  a 
reported  proUan  nor  its  possible  impact 
on  the  current  risk  assessment  withrat 
first  understanding  the  details  surround¬ 
ing  the  problem.  It  is  helpful  for  the 
FRAG  members  to  meet  periodically  to 
share  information  and  ideas. 


How  To  Document  Telephone  Interviews 


Immediately  foUowiag  a  tele{diane  inter¬ 
view,  the  FRAG  member  must  prepare  a 
narrative  summary  of  the  conversation 
and  send  it  to  the  reference  for  verifica¬ 
tion  preferably  by  certified  mail  return 
receipt  requested.  Datafaz  transmissions 
are  also  acceptable.  The  following  step  is 
exfreme/y  important.  Extra  care  must  be 
taken  to  ensure  accura^,  darity,  and 
legibility  because  these  summaries  often 
represent  the  only  written  back-up  sup¬ 
porting  the  opinicms  and  conclusions  of 
the  final  FRAG  assessment  report 
In  order  to  maintain  accurate  records 
and  facilitate  verification,  the  telephone 
record  form  should  indude  the  reference’s 
name,  full  mailing  address  and  telephone 
number,  the  date  and  time  of  the  call,  and 
the  description  of  the  contract  effort 
discussed.  A  sample  telgihone  record 
form  is  attached  at  aiqwidizD. 

Ihe  FRAG  should  send  the  telephone 
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memorandum  to  the  reference,  stating 
explidtly  that  if  the  reference  does  not 
object  to  its  content  within  the  time  sped- 
fied,  it  will  be  accepted  as  correct.  The 
amotint  of  time  allowed  for  a  response 
depends  on  the  circumstances  of  each 
procurement.  A  sample  cover  letter  is 
attached  at  appendix  E.  Note  that  the 
reference  need  not  sign  a  nondisdosure 
form  if  the  FRAG  member  withholds  the 
identity  of  the  program  and  solidtation 
number. 

If  a  reference  indicates  that  the  narra¬ 
tive  is  incorrect,  then  a  corrected  narrative 
must  be  sent  for  verification.  Experience 
indicate*  that  in  most  instances,  changes 
aremiruar.  If,  however,  a  reference  ex¬ 
presses  opposition  to  a  recmd  and  satisfac¬ 
tory  corrections  cannot  be  agreed  upon,  the 
FRAG  drould  not  rely  on  the  record. 
Another  source  may  provide  the  same 
infirrmation,  however. 


How  To  Assign  Perforbiance  Risk  Ratings 


Once  the  telephone  interviews  are  completed, 
the  entire  FRAG  needs  to  assess  all  (^erors 
and  assign  performance  risk  ratings.  The 
FRAG  should  note  of  singularly 

good  or  poor  peifonnanoe  and  relate  it  to  the 
solicitation  requirements.  Once  again,  it  is 
helpful  for  the  FRAG  to  review  the  statement 
of  work  and  specifications.  If  the  FRAG 
identifies  past  performance  problems  on  a 
prior  contract,  it  shoiild  consider  the  role 
government  fault  played  in  that  result. 

The  FRAG  should  not  limit  its  inquiry 
solely  to  the  proposing  entity  if  other  orarpo* 
rate  divisions,  oontractors  or  subcontractors 
will  perform  a  critical  element  of  the  pro* 
posed  effort.  The  performance  record  of  those 
organizations  should  be  assessed  in  accor* 
dance  with  the  solicitation.  Ferformance  risk 
assessments  should  consider  the  number  and 
severity  of  problems,  the  demonstrated 
effectiveness  of  oonrective  actions  taken  (not 


just  planned  or  promiaed),  and  the  overall 
woric  record. 

The  FRAG’s  assessment  is  usually 
baaed  upon  sulgective  judgment  It  is  not  a 
precise  or  mechanical  process.  Iheaasess- 
ment  should  include  a  description  of  the 
underlying  rationale  for  the  conclusions 
reached.  As  long  as  that  rationale  is 
reasonable,  it  will  withstand  scrutiny  even 
if  other  reasonable  conclusions  exist. 

A  word  of  caution  is  appropriate  con¬ 
cerning  offeror  pnnnises  to  correct  past 
perfimnance  failures,  as  opposed  to  actions 
already  taken  to  correct  such  failures.  A 
promise  to  improve  does  not,  by  itself, 
improve  past  performance.  However, 
demonstrated  corrective  actions  reflect  a 
commitment  to  rectify  past  performance 
problems,  and  therefore,  can  lower  the  risk 
of  jiwiilay  performance  failures. 


What  To  Include  In  The  Assessment  Report 


The  goal  is  to  avoid  saying  too  much  or  too 
little  in  the  FRAG  report.  Although  there  is 
no  need  to  restate  everything  contained  in 
the  telephone  memoranda,  the  FRAG  must 
provide  the  source  selection  authorify  with 
that  information  needed  to  make  informed 
judgments.  Conclusionaiy  statements  must 
be  supported  by  the  underlying  factual  basis. 
The  best  practice  is  to  state  the  conclusion 
and  provide  specific  examples  that  support 
that  conclusion. 

To  ensure  that  the  risk  assessments 


provide  the  necessary  background  informa¬ 
tion  and  are  structured  consistently,  the 
entire  FRAG  should  review  and  evaluate 
the  report  on  each  offeror.  During  this 
review,  the  FRAG  should  correct  state¬ 
ments  that  appear  unsupported,  inconsis¬ 
tent,  or  unnecessary. 

Occasionally  the  FRAG  will  be  unable 
to  arrive  at  a  unanimous  agreement  on  a 
particular  risk  assessment.  If  this  occurs, 
the  FRAG  may  include  the  dissenting 
opinion  as  part  of  the  assessment  report. 
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Should  The  PRAG  Brief  The  Results? 


The  PRAG’s  eubmisnoii  of  aeaeBsment  PRAG  to  eneure  that  eveiyone  fully  com* 

report  usually  completes  the  nuyor  portion  prebends  the  significance  of  the  results 
of  its  work.  However,  because  the  PRAG  being  briefed.  Experience  reveals  that 

omoept  is  relatively  new,  the  PRAG  chair  source  selection  officials  are  more  apt  to 

should  remind  the  source  selection  offi*  rely  upon  PRAG  results  if  they  thorou^dy 

dais  of  the  importance  and  purpose  of  the  understand  the  isnocesa. 


Should  PRAG  Findings  Be  Disclosed  To  Offerors? 


During  discussions  with  offeron  in  competi* 
tively  negotiated  procurements,  the  ccm- 
tracting  officer  must  disdoae  HafiriiwiriiMi 
in  the  offerors’  proposals.  Arguably, 
negative  past  p^mmance  infOTmation 
provided  by  a  reference  is  generally  not  a 
’’proposal  defidencsT  because  it  is  based 
upon  information  outside  of  that  proposaL 
Nonetheless,  a  past  perfmmance  problem 
can  be  a  significant  shortcoming  that 
must,  in  fairness,  be  brou^t  to  the  atten¬ 
tion  of  the  offeror  during  discussions  if  the 
offeror  has  not  previously  been  apprised  of 
the  problem  and  provided  an  opportunity 
to  respond. 

This  practice  validates  any  negative 
information  relied  upmi  during  the  risk 
assessment  process,  and  ensures  fairness 
for  the  competing  offerors.  The  validation 
process  is  particularly  important  when  the 
negative  information  is  provided  by  only 
one  reference,  or  when  there  is  any  doubt 


concerning  the  accuracy  of  the  informa- 
tion.  It  is  noted,  however,  that  while  the 
government  must  disclose  past  perfor¬ 
mance  problems  to  offerors’  it  need  not 
disclose  the  identity  of  its  sources. 

A  special  problem  arises  with  respect 
to  subamtractors.  Past  performance 
information  pertaining  to  a  subcontractor 
cannot  be  diwlosed  to  a  private  party 
without  the  subcontractor’s  consent 
(OFPP  Policy  Letter  92-5,  Dec.  30, 1992). 
Because  a  prime  contractor  is  a  private 
party,  the  government  needs  to  obtain  the 
subcontractor’s  consent  before  disclosing 
its  past  performance  information  to  the 
prime  during  negotiations.  'There  are  a 
variety  of  ways  to  obtain  subcontractor 
consent.  For  example,  the  solicitation 
could  require  the  prime  to  submit  its 
subcontractor’s  consent  along  with  the 
prime’s  proposal  to  the  government. 
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How  To  Treat  Past  PERFORBiANCE  Information 


PRAG  informatum  concemiiig  the  paat 
performaiice  of  an  offeror  or  of  its  pro¬ 
posed  subcontractors  should  be  treated  as 
sensitive  source  selection  information. 
This  information  sometimes  indudea 
information  that  is  {ax>prietaiy,  such  as 
trade  secrets  and  oc^dential  commercial 
or  financial  data  that  would  not  be  re¬ 
leased  tinder  the  Freedom  of  Information 
Act.  Current  laws,  regulations,  and 
policies  governing  storage,  access,  disclo¬ 
sure,  and  marking  of  source  selection  and 
proprietary  information  must  be  observed 


at  all  times.  Questions  concerning  the 
procedures  for  the  handling  of  past  perfiir- 
manoe  infimmation  should  be  referred  to 
the  contracting  officer  or  legal  counsel  fiir 
resolution. 

The  PRAG  must  retain  the  records  of 
its  activity  throus^out  the  source  selectimi 
process.  Upon  contract  award  or  cancella¬ 
tion  of  the  solicitation,  all  PRAG  records 
are  provided  to  the  contracting  <^cer  for 
retention  along  with  the  other  source 
selection  documents. 


How  To  Improve  The  PRAG  Methodology 


The  PRAG  methodology  is  a  dynamic 
process  that  will  evolve  as  our  needs 
change  and  as  our  knowledge  base  ex¬ 
pands.  It  is  important  for  all  of  us  to 
share  information  and  ideas  to  ensure 
that  this  handbook  remains  current  and 
useful. 


Each  MSG  should  establish  a  central¬ 
ised  focal  point  to  capture  and  preserve  the 
lessons  learned  firom  its  PRAGs.  Futiure 
AMC  workshops  will  call  upon  the  com¬ 
mands  to  share  their  experiences  and 
improve  this  handbook. 


IVe  owe  it  1o  our  sobUers  to  ensure  that  the  past  pei^* 
manceqf  ourcontrador^goodorbad,  isrneaittrigfid^ 
consider^  during  the  source  selectSon  process,’' 

—  Commanding  General, 
U.S,  Army  McUeriet 


Command: 


. -sy  ••<'••••< 
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Appendix 


APPENDIX  A 


Sample  Section  M  SoucnAiiON  Provision 
M.1  Evalnation  Factors  For  Award 

(a)  Selection  of  an  offeror  for  award  will  be  baaed  on  an  evaluation  of  proposals  in  three 
factors:  Technical,  Cost,  and  Perfivmance  Risk.  Each  factor  is  separately  described  below  in 
greater  detail  The  technical,  cost,  and  performance  risk  factors  will  not  be  numerically  scored 
but  rather  will  be  rated  in  an  adjectival  and  narrative  manner.  The  ultimate  objective  of  the 
evaluation  is  to  determine  which  proposal  offers  the  best  prospect  for  optimum  attainment  of 
the  objectives  of  this  program.  Negotiations  may  be  conducted  with  those  offerors  determined 
to  be  in  a  competitive  range  by  the  contracting  ofBcer. 

(b)  The  technical  factor  is  slightly  more  important  than  the  cost  factor  which  is  slightly 
more  important  than  the  performance  factor.  However,  to  be  considered  for  award  an 
offeror  must  be  determined  to  be  acceptable  in  the  tP/«hnieiil  factor.  A  deficiency  could  consti¬ 
tute  a  basis  fin*  rejection  of  a  proposal.  Award  will  be  made  to  that  offeror  whose  proposal 
represents  the  best  overall  buy  for  the  ^ovemment.  Tbe  government  reserves  the  right  to 
award  to  other  than  the  low  offeror. 

(c)  Offerors  are  urged  to  ensure  that  their  proposals  are  submitted  on  the  most  favor¬ 
able  terms  in  order  to  reflect  their  best  possible  potential,  since  less  than  the  best  potential 
could  result  in  exclusion  of  the  proposal  finom  further  consideration. 

Offerors  are  reminded  t^t  unsupported  promises  to  comply  with  the  cmitractual  re¬ 
quirements  will  not  be  sufficient.  Proposals  must  not  merely  parrot  back  the  contractual 
requirements  but  rather  must  provide  convincing  documentary  evidence  in  support  of  any 
conclusionaiy  statements  relating  to  promised  performance. 

(d)  The  offeror’s  proposal  is  presumed  to  represent  its  best  efforts  to  respond  to  the 
solicitation.  Any  inomsistency,  whether  real  or  apparent,  between  promised  performance  and 
price  should  be  explained  in  the  proposal  Unexplained  inconsistencies  resulting  from  the 
offeror’s  lack  of  understanding  of  the  nature  and  scope  of  the  work  required  may  be  grounds  for 
rejection  of  the  proposal. 

MLR  Technical  Factor 

*•* 

M.3  Cost  Factor 

*** 

M.4  Performance  Risk  Factor 

(a)  During  the  source  selection  process,  the  government  will  assess  the  relative  risks 
associated  with  each  offeror  and  proposal.  It  is  important  to  note  the  distinction  between 
proposal  risk  and  performance  ri^ 

(1)  Proposal  risks  are  those  associated  with  an  offeror’s  proposed  approach  in  meeting 
the  government’s  requirements.  Proposal  risk  is  assessed  by  the  proposal  evaluators  and  is 
integrated  into  the  rating  of  eMh  specific  evaluation  subfactor  under  the  technical  and  cost 
factors. 
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(2)  Performance  rieks  are  those  associated  mth  an  ofiferor’s  likelihood  of  success  in 
performing  the  solicitatum’s  re<iuirements  as  indicated  by  that  offeror's  record  of  past  perfinr- 
manoe.  Performance  risk  is  assessed  by  the  PHAG  and  is  assigned  a  narrative  rating  in  the 
performance  risk  factor  of  the  evaluation. 

(b)  The  government  will  conduct  a  perfixmance  risk  assessment  based  upon  the  qual* 
ily  of  the  offeror’s  past  performance  as  well  as  that  of  its  proposed  subcontractors,  as  it  relates 
to  the  probability  of  successful  accomplishment  of  the  required  effort.  When  assessing  perfor¬ 
mance  risk,  the  government  will  focus  its  inquiry  on  the  past  performance  of  the  offeror  and  its 
proposed  subcontractors  as  it  relates  to  ail  solk^tion  requirements,  such  as  cost,  schedule, 
and  performance,  including  the  ccmtractor’s  reoo^  of  conforming  to  8pecificati<ms  and  to 
standards  of  good  workmanship;  the  contractor's  record  of  ccmtaining  and  forecasting  costs  on 
any  previously  performed  cost  reimbursable  contracts;  the  contractor’s  adherence  to  contract 
schedules,  including  the  administrative  aspects  of  performance;  the  contractor’s  history  for 
reasonable  and  cooperative  behavior  and  commitment  to  customer  satisfaction;  and  generally, 
the  contractor’s  business-like  concern  for  the  interests  of  its  customers. 

(c)  A  significant  achievement,  problem,  or  lack  of  relevant  data  in  any  element  of  the 
work  can  become  an  important  consideratimi  in  the  source  selection  process.  A  negative  find¬ 
ing  under  any  element  may  result  in  an  overall  hij^  performance  rii^  rating.  Therefore, 
offerors  are  reminded  to  include  all  relevant  past  efforts,  including  demonstrated  corrective 
actions,  in  their  proposal.  The  lack  of  a  perfonnance  record  may  result  in  an  unknown  perfor¬ 
mance  risk  rating.  * 

(d)  Offerors  are  cautioned  that  in  conducting  the  perform  ance  risk  assessment,  thi. 
government  may  use  data  provided  by  the  offeror  in  its  proposal  and  data  obtained  from  other 
sources.  Since  ^e  government  may  not  necessarily  interview  all  of  the  sources  provided  by  the 
offerors,  it  is  incumbent  upon  the  offeror  to  explain  the  relevance  of  the  data  provided. 

Offerors  are  reminded  that  while  the  government  may  elect  to  consider  data  obtained  from 
other  sources,  the  biurden  of  providing  thorough  and  complete  past  performance  information 
rests  with  the  offerors. 


*  Alternatively,  the  contracting  officer  may  elect  to  state  :  "The  lack  of  a  performance 
record  may  result  in  a  high  performance  risk  rating. "  (See  page  6,  "How  to  Evaluate 
No  Past  Performance. ") 


*tbeUeve  Htat  ths.conskidtati^^gfpa^  &  hsJping  th0 

Army  make  better  Tfie  risk  assessments 

add  another  dimension  to  o^Jdl^dtif  Moid^knt^  risks  in 

an  otherwise  atttacdue  enhanoed  the 

qualitu  otournegoikd^^ 

i^theArmyl 
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Sample  Sechon  L  Soucttation  Provision 

(Caution:  Prr^iosals  that  fail  to  contain  the  information  requested  in  this  paragraph  may 
be  rejected  by  the  government) 

Performance  Risk: 

The  offerar  shall  submit  a  description  of  its  previous  government  contracts  (all  prime  and 

migoi'  subcontracts  received,  or  in  performance,  during  the  past _ years)  which  are  in  any 

way  relevant  to  the  effort  required  by  this  solicitation.  Conmercial  contracts  may  be  included 
if  necessary.  The  description  shall  include  the  following  information  in  the  following  format: 

•  Identify  in  specific  detail  for  each  previous  contract  listed,  why  or  how  you  consider  that 
effort  relevant  or  similar  to  the  effort  required  by  this  solicitation 

•  Your  (or  your  subcontractor’s)  CAG^  and  DUNNS  numbers 

•  Government  or  commercial  contracting  activify,  address,  and  telephone  number 

•  Procuring  Contracting  Officer’s  (PCO’s)  nama  and  telephone  number 

•  Government  or  commercial  contracting  activity  technical  representative,  or  COR,  and 
telephone  number 

•  Government  or  commercial  contract  administration  activify,  and  the  name  and  tele* 
phone  number  of  the  Administrative  Contracting  Officer  (ACO) 

•  Contract  number 

•  Contract  award  date 

•  Contract  type 

•  Awarded  price/cost 

•  Final,  or  projected  final,  price/cost 

•  Original  delivery  schedule 

•  Final  or  prqjected  final,  delivery  schedule 

•  A  narrative  explanation  on  each  previous  contract  listed  describing  the  objectives 
achieved  and  any  coet  growth  or  schedule  dela^  encountered.  For  any  government  contracts 
which  did  not/do  not  meet  original  requirements  with  regard  to  either  cost,  schedule,  or  techni¬ 
cal  performance,  a  brief  explanation  of  the  reason(s)  for  such  shortcomings  and  any  demon¬ 
strated  corrective  actions  taken  to  avoid  recurrence.  The  offeror  shall  also  provide  a  copy  of 
any  cure  notices  or  show  cause  letters  received  on  each  previous  contract  listed  and  a  descrip¬ 
tion  of  any  corrective  action  by  the  offeror  or  proposed  subcontractor. 

•  The  offeror  shall  also  provide  the  above  required  infonnation  for  any  and  all  contracts  it 

has  had  terminated  in  whole  or  in  part,  for  default  during  the  past _ years,  to  include  those 

ciirrently  in  the  process  of  such  termination  as  well  as  those  which  are  not  sixnilar  to  the 
proposed  effort  The  contractor  shall  list  each  time  the  delivery  schedule  was  revised  and 
provide  an  explanation  of  why  the  revision  was  necessary. 
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*  New  corporate  entities  may  submit  data  on  prior  contracts  involving  its  officers  and 
employees.  However,  in  addition  to  the  other  requirements  in  this  section,  the  c^eror  nhall 
discuss  in  detail  the  role  performed  by  such  persons  in  the  prior  ctmtracts  dted. 

*  Offerors  shall  provide  an  outline  of  how  the  effort  required  by  the  solicitation  will  be 
assigned  for  performance  within  the  contractor’s  corporate  entity  and  among  proposed  subcon¬ 
tractors.  Information  required  in  the  above  paragraphs  shall  be  provided  for  each  pn^osed 
subcontractor  who  will  perform  a  significaiit  portion  of  the  effiirt  “Significant”  is  defined  for 
these  purposes  in  terms  of  estimated  dollar  amount  of  the  suhcontract  (e.g.,  $1,000,000  or 
more)  and/or  in  terms  of  criticality  of  the  subcontracted  work  to  the  whole.  With  regard  to 
prime  contract  assignments  that  will  be  performed  by  you  and  not  a  proposed  subcontractor, 
you  shall  indicate: 

1)  what  internal  corporate  bodies/divisions  will  accomplish  which  portions  of  the  effort, 

2)  whether  or  not  those  divisions  were  responsible  for  performance  under  the  previous  con¬ 
tracts  dted  for  the  instant  proposal,  and 

3)  if  those  divisions  have  relocated  since  the  accomplishment  of  previous  dted  contract  efforts, 
a  description  of  any  changes  arising  from  that  relocation  in  terms  of  key  personnel,  facilities 
and  equipment. 

*  Offerors  shall  indude  in  their  proposal  the  written  consent  of  their  proposed  significant 
subcontractors  to  allow  the  government  to  discuss  the  subcontractor’s  past  performance  evalu¬ 
ation  with  the  ofieror  during  negotiations. 

Note:  Offerors  are  reminded  that  both  independent  data  and  data  provided  by  offerors  in 
their  proposals  may  be  used  to  evaluate  offeror  past  performance.  Since  the  government  may 
not  necessarily  interview  all  of  the  sources  provided  by  the  offerors,  it  is  incumbent  upon  the 
offeror  to  explain  the  relevance  of  the  data  provided.  The  government  does  not  assume  the 
duty  to  search  for  data  to  cure  problems  it  finds  in  proposals.  The  burden  of  providing  thor¬ 
ough  and  complete  past  performance  information  remains  with  the  offerors.  Proposals  that  do 
not  contain  the  information  requested  by  this  paragraph  risk  rejection  or  high  risk  rating  by 
the  government. 


“The  purpose  ofPRAG  is  to  evaluate  peiformance  risk  — 
to  seek  out  and  reward  good  petfonnance.* 

— *  Deputy  Xktmmtmding  General 
cr*S«  Army^m^eriel  Command 
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Sample  PRAG  Assessmeot  Plan 


Definitioiis 

Perfisrmanoe  Ri$k$: 

Performance  rieke  are  those  associated  with  an  offeror's  likelihood  of  success  in  perform* 
ing  the  solicitatian's  requirements  as  indicated  by  that  offeror's  record  of  past  performance. 
Performance  risk  is  assessed  by  the  Performance  Bisk  Assessment  Group  (PRAG)  and  is 
assigned  a  narrative  rating  in  the  Performance  Sisk  Factor  of  the  evaluation. 

Perfimnance  Ruk  Asaeaament  Orot^: 

A  PRAG  is  the  source  selection  entity  that  asaenes  performance  risk.  The  PRAG  may 
either  be  separate  from  the  SSEB  and  report  directly  to  the  SSAC,  or  operate  as  a  separate 
group  within  the  SSEB  and  report  through  the  SSEB  chairperson  to  the  SSAC.  Each  contract* 
ing  activity  determines  the  appropriate  compositiQa  and  structure  of  its  PRAGs,  depending 
upon  the  siae,  nature,  and  complexity  of  a  particular  procurement. 

Proposal  Riaks: 

Proposal  risks  are  those  associated  with  an  offeror's  proposed  approach  in  meeting  the 
government's  requirements.  Proposal  risk  is  assessed  by  the  proposal  evaluators  and  is  into* 
grated  into  the  rating  of  eadi  specific  evaluation  subfactor  under  the  technical  and  cost  factors. 

The  Performance  Risk  Assessment  Group 

Responsibilities: 

The  PRAG  shall  perform  an  in*depth  review  and  evaluation  of  the  performance  data 
provided  by  offerors  and  obtained  finom  other  sources  to; 

•  Assess  each  offeror's  past  and  current  perfonnance  as  it  relates  to  the  solicitation 
requirements.  The  PRAG  should  consider  the  relevancy,  recency  and  accura^  of  the  data  in 
arriving  at  its  overall  assessment. 

*  Identity  strong  «wH  weak  points  for  use  during  negotiations  and/or  contract  administra* 

tion. 


Performance  Risk  Assessment: 

The  perfcmnance  risk  assessment  conducted  by  the  PRAG  assesses  each  offeror’s  record  of 
performance  to  determine  the  offeror’s  likelihood  sucoms  in  performing  the  required  efiint. 
The  PRAG  must  focus  its  inquiry  on  the  offeror’s  reccard  of  performance  as  it  relates  to  the 
performance  of  the  solicitation  requirements.  Therefore,  the  PRAG  must  become  thorou^y 
familiar  with  the  statement  of  wwk  and  specifications.  Since  the  PRAG  does  not  perform  the 
proposal  risk  assessment  (the  SSEB’s  proposal  evaluators  do  that),  it  does  not  normally  review 
the  offerors  proposals. 

The  PRAG’s  performance  risk  assessment  is  not  solely  limited  to  the  prime  contractor 
division  submittixv  the  proposal  when  other  divisUms,  corporate  entities,  critical  subcontrac* 
tors,  or  teaming  contractors  pwform  a  critical  element  of  ^required  effort  In  such  cases,  the 
PRAG  should  evaluate  the  other  organisation’s  performance  record. 
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Each  performance  risk  aaaesement  will  cosiaider  the  number  and  severity  of  problems,  the 
effectiveness  of  corrective  actions  taken,  and  the  overall  woric  record.  The  assessment  c£ 
performance  risk  is  not  intended  to  be  the  product  of  a  mwchaniftwl  or  mathematical  analysis  of 
an  offeror’s  performance  on  a  list  of  contrail,  but  rather  the  product  of  subjective  judgment  of 
the  PRAG  a^r  it  considers  all  available,  releimnt  and  recent  in&nnation.  The  following 
definitions  of  performance  risk  should  be  used: 

•  High  Performance  Risk  -  Based  on  the  offeror’s  perfimmance  record,  significant  doubt 
exists  that  the  offeror  will  successfully  perfbnn  the  required  effort. 

•  Moderate  Performance  Risk  -  Based  on  the  offeror’s  performance  record,  smne  doubt 
exists  that  the  offeror  will  successfully  perform  the  required  effort. 

•  Low  Performance  Risk  -  Based  on  the  offeror’s  perfonnance  record,  little  doubt  exists 
that  the  offeror  will  successfully  perform  the  required  effort. 

•  Unknown  Performance  Risk  -  No  performance  record  identifiable  (this  category  is 
optional). 

Note:  Each  of  the  high,  moderate,  and  low  risk  categories  may  be  further  subdivided  to 
enhance  the  comparative  analysis  of  offerors. 

[Insert  Section  M  of  the  Solicitation  here] 

Documentation: 

•  The  FRAG’s  performance  risk  assessment  report  will  be  provided  directly  to  the  con* 
tracting  officer.  The  results  of  the  PRAG  evaluation  will  also  be  briefed  to  the  contracting 
officer  by  the  PRAG  chairperson. 

•  The  PRAG’s  documentation  and  presentations  should  address  the  following: 

—  The  sources  of  the  perfonnance  data 

—  The  relevancy  of  the  data  to  the  program  requirements 

—  The  cunency  of  the  data 

—  The  performance  risk  assessment  of  each  offeror 

—  The  supporting  rationale  for  each  performance  risk  assessment 

—  The  strong  and  weak  areas  of  each  offeror  for  use  during  negotiations  and/or  con¬ 
tract  administration 

—  Any  other  matters  deemed  relevant 
Gathering  Performance  Data 

•  The  two  primary  sources  of  perfonnance  data  are  the  contractor  references  contained  in 
the  performance  risk  volume  of  the  offeror’s  proposal  and  the  AMC  CIS. 
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•  IheCISdoesnotoontamasulifjectiveanalsrBisofpastperfinrmaiice.  Instead,  it  was 
designed  to  provide  the  PRAG  with  the  keyfiictual  information  needed  to  commence  an  inves¬ 
tigation  into  the  ctmtiactor’s  perfonnance  history.  The  actual  assessment  has  been  reserved 
for  the  PRAG  members  who  can  best  determine  which  information  is  most  relevant  to  the 
acquisition. 

•  Upon  request,  the  CIS  will  provide  the  PRAG  with  a  Contractor  Information  Report 
(CIR)  on  an  ofTeror’s  contract  hist^  within  AMC.  In  reviewing  that  data  report,  the  PRAG 
should  exerdse  its  own  judgment  to  determine  which  of  the  (rfferor’s  past  efforts  are  most 
relevant  to  the  solicitation’s  requirements.  Key  points  of  contact  will  be  identified  on  the  data 
report  for  direct  telephonic  contact 

•  The  PRAG  will  obtain  ^diatever  information  it  deems  most  relevant  to  the  required 
effort  by  telephonic  and/or  written  inquiry  with  the  points  of  contact  identified  on  the  CIR.  It 
is  important  that  each  discussion  be  accurately  summarized  on  a  PRAG  Telephone  Interview 
Report  Form  for  it  is  this  material  which  will  later  serve  as  back-up  for  the  PRAG’s  perfor¬ 
mance  risk  assessment  A  copy  of  the  Telephone  Interview  Report  Form  must  he  promptly  eent 
by  certified  mail  (return  receipt  requested)  or  by  data  fax  to  the  point  of  contact  /br  verification. 
There  is  usually  no  need  to  divulge  the  s^dtation  number,  or  other  identifying  information  to 
the  reference.  Ifyou  do  so,  however,  you  need  to  obtain  a  nondisdosure  statement 

•  The  PRAG  should  also  ezerdse  its  judgment  in  determining  which,  if  any,  of  the  con¬ 
tractor  supplied  references  should  be  called  for  additional  information  or  verification.  Both 
negative  and  positive  information  should  be  carroborated  before  it  is  relied  upon  to  any  signifi¬ 
cant  degree  to  ensure  accuracy  in  the  final  PRAG  report  and  fairness  in  the  overall  process. 
PRAG  Telephone  Interview  Report  Forms  should  be  completed  for  these  contacts  as  well. 

•  The  key  to  the  success  of  each  performance  risk  assessment  is  the  PRAG’s  willingness 
and  ability  to  seek  out  the  moet  relevant,  recent.  And  accurate  information  available.  Should  a 
PRAG  member  be  unable  to  obtain  information  for  a  reference,  he  or  she  may  contact  the 
PRAG  chairperson  who  should  seek  assistance  through  the  sotirce  selection  hierarchy. 


7  was  the  contractlr^  qfficenJPiRAG  was  a  new  initiative 
at  ptat^ihne^Tp^d  H  and  it  work^  The  (xmtractar  deliv¬ 
ered  d^%tdii^pnjdud^dhd^  The 

ttsers  ttfere  thrili^w^U  and  U  also  played  a  rncdor  role 
ihDe^ 

^  Army  OommunicatUms- 

station 
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Sample  Format  For  Telephone  Interview  Report 


PRAGI.D.  NUMBER: 

CONTRACTOR:  (Name  &  Address) 

PERSON  CONTACTED:  (Name.  Address.  Phone  «) 

DATE  &  TIME  OF  CONTACT: 

lam  (home).  My  telephone  number  is  (#).  I  am  calling  in  reference  to  contrac- 
tor  (name).  My  questions  will  pertain  to  that  contractor’s  record  ci past  and  current  pexfiar- 
mance.  The  information  that  you  provide  will  be  uaed  in  the  awarding  of  federal  contracts. 
Therefore  it  is  important  that  your  information  be  as  factual  and  acciirate  as  possible.  A 
summaiy  of  thin  discussion  will  be  sent  to  you  for  jfour  records.  If  that  summary  is  inaccurate 
or  incomplete  in  any  way.  please  contact  me  immediately. 

SUMMARY  OF  DISCUSSION: 


Telephone  # 


Signature  of  FRAG  Member 


Reminders  for  PRAG  Member: 

•  Disctiss  recency  and  relevance  of  information 

•  Read  summazy  to  person  contacted 

•  Send  copy  to  person  contacted 

•  Withhold  the  identity  of  your  program  and  solicitation  number,  if  practicable,  to  avoid 
having  to  obtain  a  mm-discloszire  statement  from  the  person  contacted 
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Sampi£  Format  For  Telefbone  Interview  Report  Cover  Letter 


[On  letteHuadJ 


Address 


1.  Attached  is  a  summaiyofyourteleidume  conversation  with  a  member  of  the  Perfor- 
mance  Risk  Assessment  Group  on  (date). 

2.  If  this  ofiBce  does  not  hear  from  you  by  we  will  assume  that  the  summary  of  the 
discussion  is  correct  If  you  have  any  questions  or  cmnments  you  may  address  them  to 
(address).  You  may  also  call  me  directly  at  (pfumeM). 

3.  We  thank  you  for  your  time  and  awMutiiwfto  regarding  thi«  effort. 


End.  Chairman,  PRAG 
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Sample  Quesiions  And  Ideas  F6ft  T^upbone  Interviews  And 

Questionnaires 

*  Becaiise  the  word  **PRAG**  ifl  new,  it  may  not  be  reoQgnued  by  the  references.  To  avoid 
confusion,  simply  state  that  the  contractor’s  past  performance  is  being  assened  for  a  source 
selection.  There  is  iisually  no  need  to  divulge  the  solicitation  number,  program  deacription,  or 
other  identifying  informati<m  to  the  reference.  If  you  do  so,  you  need  to  obtain  a  nondimloBure 
statement. 

•  Confirm  the  following  data  received  from  the  contractor: 

—  Contract  number  and  type 

—  Award  amount  and  final  or  projected  final  amount 
—  Award  delivery  schedule  and  final  or  projected  fiwl  delivery  schedule 
—  Nature  of  the  effort  (i.e.,  the  scope  of  the  effort,  the  types  of  tasks  involved  and  the 
product  to  be  delivered) 

•  If  the  award  amount  or  delivery  schedule  has  changed,  find  out  what  caused  the  change. 

*  Discover  what  role  the  reference  played  (e.g.,  COR,  contract  specialist,  ACO,  etc.)  and 
for  how  long. 

•  If  a  problem  is  uncovered,  discuss  what  the  government  and  contractor  did  to  resolve  it. 

*  Ask  for  a  description  of  the  types  of  personnel  (skills  and  expertise)  the  contractor  used 
and  the  overall  quality  of  the  contractor’s  team.  Did  the  company  appear  to  use  personnel 
with  the  appropriate  skills  and  expertise? 

*  Ask  how  the  contractor  performed  considering  quality  of  perfcnmance,  responsiveness, 
schedule,  overall  management,  technical  performance,  and  financial/cost  management 

*  Ask  whether  the  contractor  was  cooperative  in  negotiations  and  in  resolving  issues. 

•  Inquire  whether  there  were  any  particularly  significant  risks  involved  in  performance  of 
the  effort. 

*  Ask  if  the  company  appeared  to  apply  sufficient  resources  (personnel  and  facilities)  to 
the  effort. 

•  Ask  if  the  company  used  subcontractors.  If  so,  what  was  the  relationship  between  the 
prime  and  the  subcontractors?  What  was  the  management  rde  of  the  inime  and  how  well  did 
it  manage  the  subcontractors?  Did  the  subcontractme  perform  the  bulk  of  the  effort  or  just  add 
breadth  or  depth  on  particular  technical  areas?  If  the  subcontractors  worked  on  specific  tech¬ 
nical  areas,  what  were  those  areas  and  why  were  they  accomplished  by  the  subcontractors 
rather  than  the  prime? 

*  If  a  problem  is  uncovered  that  the  reference  is  nnfamiliAr  with,  ask  for  another  indi¬ 
vidual  who  might  have  the  information. 

*  Inquire  whether  there  are  other  past  efforts  by  this  firm  with  the  reference’s  agency. 

•  Inquire  what  the  company’s  strong  points  are  or  what  the  reference  liked  the  most 
about  them. 

•  Inquire  what  the  company’s  weak  points  are  or  what  the  reference  liked  least  about 
them. 

■  Inquire  whether  the  reference  has  any  reservations  about  recommending  a  future 
contract  award  to  this  company. 

•  Inquire  whether  the  reference  knows  of  anyone  else  iriio  might  have  past  performance 
information  on  the  offeror. 
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APPENDIX  G 

Sampub  Questionnaibe 


Program  Name: 

Contract  No.: 

1.  Please  specify  the  contract  requirements,  purpose,  and  technology. 


2.  Did  the  contractor  meet  the  original  equipment  performance  requirements?  Please 
explain: 


3.  Did  the  contractor  request  specification  relief?  If  so,  was  there  an  impact  on  system 
performance,  cost  or  deliveiy? 


4.  Did  the  contractor  use  Ada  language?  If  yes,  did  the  contractor  meet  the  Ada  language 
and  software  requirements?  Please  explain: 


5.  Did  the  contractor  meet  test  schedule  requirements?  Please  explain: 

6.  Were  any  Qualify  Deficiency  Reports  (QDR)  or  corrective  action  requests  submitted  to 
the  contractor  due  to  qualify  deficiencies?  Please  explain: 

7.  Opinion:  Quality,  reliability  and  maintainabilify  of  equipment  delivered.  Very  Good  ( ) 
Good  ( )  Acceptable  ( )  Marginally  Acceptable  ( )  Please  explain: 


8.  Was  the  contractor’s  engineering  maniigwTngnt  effective  in  controlling  costs,  schedule 
and  performance  requirements?  Please  explain: 


9.  Did  the  contractor  successfully  manage  its  subcontractors?  Please  explain: 


10.  Was  human  engineering^anprint  a  requirement?  If  so,  was  it  satisfactoiy?  Please 
explain: 

11.  Was  logistics  support  satisfactoiy  in  meeting  omtract  requirements?  Please  explain: 
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12.  At  completion  of  the  contract,  waa  the  contractor  committed  to  custmner  satiafactum? 
Pleaae  explain: 


13.  Rate  the  contractor  a  overall  technical  performance:  GoodO  FairO  PoorO  Would 
you  recommend  this  contractor  for  other  contracts?  Please  explain: 


14.  Were  there  any  problems  with  Engineering  Change  Proposal,  Requests  for  Waivers,  or 
Requests  for  Deviations?  Please  explain: 


15.  Were  there  any  problems  with  Logistics  Support  Documentation?  Pleaae  explain: 


16.  During  technical  meetings,  was  the  contractor  cooperative  and  receptive  to  govern¬ 
ment  concerns  affecting  production  and/or  perfinmance  requirements?  Please  explain: 


17.  With  respect  to  design,  engineering  rapability,  and  overall  technical  performance, 
would  you  recommend  this  contractor  for  similar  government  contracts?  Please  explain: 


18.  How  would  you  rate  the  contractor’s  technical  performance  on  this  contract? 
Outstanding  ( )  Good  ( )  Poor  ( )  Please  explain: 


19.  Do  you  know  of  anyone  else  who  might  have  relevant  information  concerning  this 
contractor’s  past  performance?  Please  mcplain: 


20.  Please  make  any  additional  cmnments  you  wish  here: 


The  proponent  at  this  pamphlet  is  the  U.S.  Anny  Materiel  Conwnand.  Users  are  invited 
to  send  comments  and  suggested  improvements  on  DA  Form  2028  (Recommeded 
Changes  to  Publications  and  Blank  Forms)  to  the  Commander,  USAMC,  ATTN: 
AMCAQ,  5001  Eisenhowrer  Avenue,  Alexandria,  VA  22333-0001. 
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